[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 664510341 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002395] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.006912] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009195] LSM: Security Framework initializing [ 0.010063] Yama: becoming mindful. [ 0.011065] SELinux: Initializing. [ 0.012088] *** VALIDATE selinux *** [ 0.021211] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027319] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028281] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029181] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031148] *** VALIDATE tmpfs *** [ 0.032458] *** VALIDATE proc *** [ 0.033274] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036222] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041688] debug: unmapping init [mem 0xffffffffa3259000-0xffffffffa3260fff] [ 0.043342] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044978] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053288] rcu: Hierarchical SRCU implementation. [ 0.055651] smp: Bringing up secondary CPUs ... [ 0.056720] x86: Booting SMP configuration: [ 0.057029] .... node #0, CPUs: #1 #2 #3 [ 0.066163] smp: Brought up 1 node, 4 CPUs [ 0.068033] smpboot: Max logical packages: 1 [ 0.069016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128689] node 0 deferred pages initialised in 58ms [ 0.133405] devtmpfs: initialized [ 0.134330] x86/mm: Memory block size: 128MB [ 0.138170] gcov: version magic: 0x41383552 [ 0.140418] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142461] pinctrl core: initialized pinctrl subsystem [ 0.143222] [ 0.143973] ************************************************************* [ 0.144021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145088] ** ** [ 0.146010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147011] ** ** [ 0.148018] ** This means that this kernel is built to expose internal ** [ 0.149013] ** IOMMU data structures, which may compromise security on ** [ 0.150012] ** your system. ** [ 0.151012] ** ** [ 0.152091] ** If you see this message and you are not debugging the ** [ 0.153012] ** kernel, report this immediately to your vendor! ** [ 0.154011] ** ** [ 0.155014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156015] ************************************************************* [ 0.157814] NET: Registered protocol family 16 [ 0.160805] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174135] cpuidle: using governor menu [ 0.175846] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178540] PCI: Using configuration type 1 for base access [ 0.181129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197040] cryptd: max_cpu_qlen set to 1000 [ 0.199859] ACPI: Added _OSI(Module Device) [ 0.200000] ACPI: Added _OSI(Processor Device) [ 0.200000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.203012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212229] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220000] ACPI: Interpreter enabled [ 0.223071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226020] ACPI: Using IOAPIC for interrupt routing [ 0.228326] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235073] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259793] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283718] acpiphp: Slot [2] registered [ 0.285135] acpiphp: Slot [5] registered [ 0.288149] acpiphp: Slot [6] registered [ 0.291139] acpiphp: Slot [7] registered [ 0.294137] acpiphp: Slot [8] registered [ 0.297244] acpiphp: Slot [9] registered [ 0.299870] acpiphp: Slot [10] registered [ 0.303141] acpiphp: Slot [3] registered [ 0.305133] acpiphp: Slot [4] registered [ 0.306662] acpiphp: Slot [11] registered [ 0.309240] acpiphp: Slot [12] registered [ 0.312208] acpiphp: Slot [13] registered [ 0.314219] acpiphp: Slot [14] registered [ 0.316176] acpiphp: Slot [15] registered [ 0.318195] acpiphp: Slot [16] registered [ 0.320132] acpiphp: Slot [17] registered [ 0.322125] acpiphp: Slot [18] registered [ 0.324329] acpiphp: Slot [19] registered [ 0.326485] acpiphp: Slot [20] registered [ 0.328371] acpiphp: Slot [21] registered [ 0.331154] acpiphp: Slot [22] registered [ 0.333214] acpiphp: Slot [23] registered [ 0.335128] acpiphp: Slot [24] registered [ 0.337135] acpiphp: Slot [25] registered [ 0.339199] acpiphp: Slot [26] registered [ 0.342119] acpiphp: Slot [27] registered [ 0.343220] acpiphp: Slot [28] registered [ 0.346124] acpiphp: Slot [29] registered [ 0.348112] acpiphp: Slot [30] registered [ 0.350186] acpiphp: Slot [31] registered [ 0.352142] PCI host bridge to bus 0000:00 [ 0.354019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.356025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.359023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.362026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.367025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.371026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.375182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.378127] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.385368] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.400022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.407039] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.414021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.419021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.423038] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.428623] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.434581] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.439162] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.444712] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.455019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.476015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.483014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.494066] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.509015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.521016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.551033] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.586723] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.607016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.631017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.678017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.708000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.727019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.741014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.782015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.794176] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.808018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.817016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.840016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.860221] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.871015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.884066] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.912018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.927178] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.936016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.951015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.969000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.983637] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.986490] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.989377] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.991475] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.994240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.999060] iommu: Default domain type: Passthrough [ 1.001449] SCSI subsystem initialized [ 1.003181] ACPI: bus type USB registered [ 1.004215] usbcore: registered new interface driver usbfs [ 1.007093] usbcore: registered new interface driver hub [ 1.009071] usbcore: registered new device driver usb [ 1.011238] pps_core: LinuxPPS API ver. 1 registered [ 1.013011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.017056] PTP clock support registered [ 1.019483] EDAC MC: Ver: 3.0.0 [ 1.020000] PCI: Using ACPI for IRQ routing [ 1.022476] NetLabel: Initializing [ 1.025010] NetLabel: domain hash size = 128 [ 1.027017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.030178] NetLabel: unlabeled traffic allowed by default [ 1.033362] vgaarb: loaded [ 1.035129] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.037015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.045281] clocksource: Switched to clocksource kvm-clock [ 1.207401] VFS: Disk quotas dquot_6.6.0 [ 1.210620] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.215335] *** VALIDATE ramfs *** [ 1.217013] *** VALIDATE hugetlbfs *** [ 1.219896] pnp: PnP ACPI init [ 1.225235] pnp: PnP ACPI: found 6 devices [ 1.249343] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.254174] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.257694] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.261110] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.264304] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.269186] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.273804] NET: Registered protocol family 2 [ 1.277721] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.286164] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.294135] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.302714] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.307860] TCP: Hash tables configured (established 65536 bind 65536) [ 1.312784] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.317408] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.322337] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.327210] NET: Registered protocol family 1 [ 1.331100] RPC: Registered named UNIX socket transport module. [ 1.334667] RPC: Registered udp transport module. [ 1.336505] RPC: Registered tcp transport module. [ 1.338430] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.342076] NET: Registered protocol family 44 [ 1.344212] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.348738] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.351201] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.353513] PCI: CLS 0 bytes, default 64 [ 1.358335] Unpacking initramfs... [ 3.943714] debug: unmapping init [mem 0xffff9760bcc54000-0xffff9760bffbffff] [ 3.947745] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.949713] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.952315] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.740871] Initialise system trusted keyrings [ 4.744706] Key type blacklist registered [ 4.747785] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.763256] zbud: loaded [ 4.768812] *** VALIDATE nfs *** [ 4.770360] *** VALIDATE nfs4 *** [ 4.771965] pstore: using deflate compression [ 4.779932] Platform Keyring initialized [ 4.953591] NET: Registered protocol family 38 [ 4.956155] Key type asymmetric registered [ 4.958119] Asymmetric key parser 'x509' registered [ 4.960803] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.965306] io scheduler mq-deadline registered [ 4.967435] io scheduler kyber registered [ 4.969563] io scheduler bfq registered [ 4.972330] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.976365] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.979451] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.985219] ACPI: Power Button [PWRF] [ 4.992262] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.000611] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.031382] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.055216] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.087089] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.118711] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.148975] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.153602] Non-volatile memory driver v1.3 [ 5.155395] Linux agpgart interface v0.103 [ 5.190251] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.193114] vda: detected capacity change from 0 to 74686464 [ 5.233220] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.243450] vdb: detected capacity change from 0 to 1073741824 [ 5.292499] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.303461] vdc: detected capacity change from 0 to 2621440000 [ 5.343702] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.350424] vdd: detected capacity change from 0 to 2621440000 [ 5.391370] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.398533] vde: detected capacity change from 0 to 4294967296 [ 5.444274] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.453296] vdf: detected capacity change from 0 to 4294967296 [ 5.468887] libphy: Fixed MDIO Bus: probed [ 5.478418] usbcore: registered new interface driver usbserial_generic [ 5.483633] usbserial: USB Serial support registered for generic [ 5.488535] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.493678] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.496099] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.499996] mousedev: PS/2 mouse device common for all mice [ 5.507506] rtc_cmos 00:05: RTC can wake from S4 [ 5.512372] rtc_cmos 00:05: registered as rtc0 [ 5.514394] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.515324] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.530799] intel_pstate: CPU model not supported [ 5.535751] hid: raw HID events driver (C) Jiri Kosina [ 5.540749] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.545091] usbcore: registered new interface driver usbhid [ 5.546385] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.547710] usbhid: USB HID core driver [ 5.548917] drop_monitor: Initializing network drop monitor service [ 5.555736] Initializing XFRM netlink socket [ 5.558077] NET: Registered protocol family 10 [ 5.561282] Segment Routing with IPv6 [ 5.562759] NET: Registered protocol family 17 [ 5.564451] mpls_gso: MPLS GSO support [ 5.569964] RAS: Correctable Errors collector initialized. [ 5.572095] AVX version of gcm_enc/dec engaged. [ 5.574480] AES CTR mode by8 optimization enabled [ 5.654226] sched_clock: Marking stable (5654080806, 0)->(7027523577, -1373442771) [ 5.657895] registered taskstats version 1 [ 5.660282] Loading compiled-in X.509 certificates [ 5.662712] zswap: loaded using pool lzo/zbud [ 5.692518] Key type big_key registered [ 5.707705] Key type encrypted registered [ 5.709555] ima: No TPM chip found, activating TPM-bypass! [ 5.714832] ima: Allocated hash algorithm: sha1 [ 5.716727] ima: No architecture policies found [ 5.718932] evm: Initialising EVM extended attributes: [ 5.724082] evm: security.selinux [ 5.727554] evm: security.ima [ 5.729651] evm: security.capability [ 5.733939] evm: HMAC attrs: 0x1 [ 5.739452] rtc_cmos 00:05: setting system clock to 2026-09-02 14:15:21 UTC (1788358521) [ 5.753149] debug: unmapping init [mem 0xffffffffa4203000-0xffffffffa43fffff] [ 5.757517] debug: unmapping init [mem 0xffffffffa2f82000-0xffffffffa3258fff] [ 5.767940] Write protecting the kernel read-only data: 28672k [ 5.772788] debug: unmapping init [mem 0xffffffffa1603000-0xffffffffa17fffff] [ 5.777420] debug: unmapping init [mem 0xffffffffa1f14000-0xffffffffa1ffffff] [ 5.830486] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.842570] systemd[1]: Detected virtualization kvm. [ 5.844182] systemd[1]: Detected architecture x86-64. [ 5.848122] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.877534] systemd[1]: No hostname configured. [ 5.879961] systemd[1]: Set hostname to . [ 5.883366] random: systemd: uninitialized urandom read (16 bytes read) [ 5.887435] systemd[1]: Initializing machine ID from random generator. [ 6.042201] random: ln: uninitialized urandom read (6 bytes read) [ 6.306053] random: systemd: uninitialized urandom read (16 bytes read) [ 6.312199] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.336789] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.363739] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.061120] device-mapper: uevent: version 1.0.3 [ 8.063983] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.191707] random: fast init done [ 9.198415] virtio_net virtio0 ens2: renamed from eth0 [ 9.419396] scsi host0: ata_piix [ 9.705137] scsi host1: ata_piix [ 9.709946] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.714966] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.255952] random: crng init done [ 14.261930] random: 7 urandom warning(s) missed due to ratelimiting [ 16.906931] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.588969] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.400315] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.860138] SELinux: Disabled at runtime. [ 21.949291] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.962789] systemd[1]: Detected virtualization kvm. [ 21.965935] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.150352] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.157311] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.170775] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.180401] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.185987] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.209990] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.232248] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ 23.736902] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ 24.228102] hrtimer: interrupt took 3094030 ns [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 25.196288] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.590864] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.743456] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.616241] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.815128] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.870876] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.613178] NFS: Registering the id_resolver key type [ 32.615843] Key type id_resolver registered [ 32.618671] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg402-server login: [ 65.941439] libcfs: loading out-of-tree module taints kernel. [ 65.962531] Key type ._llcrypt registered [ 65.964070] Key type .llcrypt registered [ 66.019841] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 74.505296] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 75.202885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.208447] alg: No test for adler32 (adler32-zlib) [ 76.227066] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 76.538969] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 78.151169] Key type lgssc registered [ 78.775231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.357206] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 102.852930] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 108.053553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 108.062631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 109.174170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 109.189113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 109.232813] Lustre: lustre-MDT0000: new disk, initializing [ 109.273690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 109.281689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 110.719557] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.810284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 116.863759] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 116.886462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 116.890264] Lustre: Skipped 1 previous similar message [ 116.937818] Lustre: lustre-MDT0001: new disk, initializing [ 116.972376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 116.983428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 116.988033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 118.659245] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.355291] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 125.493910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 125.592763] Lustre: lustre-OST0000: new disk, initializing [ 125.594540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 125.596942] Lustre: 8420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 125.621213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 127.988783] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.031103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 130.034558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 130.059850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 134.299804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 134.350067] Lustre: lustre-OST0001: new disk, initializing [ 134.353034] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 134.355930] Lustre: 9492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 134.378683] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 136.703195] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.264307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 137.270774] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 137.301484] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 142.870527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 147.706816] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 150.253348] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing check_logdir /tmp/testlogs/ [ 152.228884] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing yml_node [ 154.124836] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 155.165935] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 156.240873] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 156.942466] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 2 10:17:52 EDT 2026 [ 163.616815] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 164.261308] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 178.144052] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 178.144584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 178.148844] Lustre: Skipped 3 previous similar messages [ 178.151925] Lustre: Skipped 2 previous similar messages [ 182.458736] Lustre: server umount lustre-MDT0000 complete [ 184.155096] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358699 with bad export cookie 5595489207568063995 [ 184.158097] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 184.162246] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 188.384145] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 188.384271] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 188.384506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 188.384511] Lustre: Skipped 1 previous similar message [ 188.392467] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 188.398692] Lustre: Skipped 1 previous similar message [ 189.407991] LustreError: 10095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 189.408233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 189.415975] LustreError: 10095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 189.418767] Lustre: Skipped 2 previous similar messages [ 190.352287] Lustre: server umount lustre-MDT0001 complete [ 192.079816] Lustre: server umount lustre-OST0000 complete [ 193.839133] Lustre: server umount lustre-OST0001 complete [ 196.576575] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 200.679713] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 214.833726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.943861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.958575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 215.000259] Lustre: lustre-MDT0000: new disk, initializing [ 215.039796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.047775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 216.667836] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 221.509430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.562578] Lustre: 15035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 221.701325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 223.314892] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 227.975721] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 230.346751] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 233.329094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.464249] Lustre: lustre-OST0000: new disk, initializing [ 233.467040] Lustre: Skipped 1 previous similar message [ 233.469228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 233.471721] Lustre: Skipped 2 previous similar messages [ 233.474603] Lustre: 17001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.515436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.518296] Lustre: Skipped 1 previous similar message [ 234.889219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.898740] Lustre: Skipped 1 previous similar message [ 234.902674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.953102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.946195] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 240.175993] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 241.776266] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 241.862096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 243.487351] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 243.573303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 245.217370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 245.217675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.220903] LustreError: Skipped 1 previous similar message [ 245.227699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 245.229798] Lustre: Skipped 1 previous similar message [ 250.338231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 250.340555] Lustre: Skipped 1 previous similar message [ 250.392709] Lustre: server umount lustre-OST0000 complete [ 252.383956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 252.386936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.391280] Lustre: Skipped 1 previous similar message [ 260.576312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 260.579486] Lustre: Skipped 4 previous similar messages [ 266.207154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 266.276536] Lustre: server umount lustre-MDT0000 complete [ 267.631366] LustreError: 15025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358783 with bad export cookie 5595489207568065528 [ 267.633537] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.635122] LustreError: 15025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 267.760757] Lustre: server umount lustre-MDT0001 complete [ 269.322792] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:19:44 (1788358784) [ 270.162186] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 270.770493] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:19:46 (1788358786) [ 273.262482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.461745] LustreError: 19411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 273.499354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 275.051704] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 278.036561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.550687] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 281.923849] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 283.813420] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 285.961684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 288.037755] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 290.674693] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 308.708068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 308.708791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.709887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 308.709897] Lustre: Skipped 2 previous similar messages [ 308.717214] Lustre: Skipped 2 previous similar messages [ 310.677300] Lustre: server umount lustre-OST0000 complete [ 318.293142] Lustre: server umount lustre-MDT0000 complete [ 318.944898] LustreError: 19430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.950331] LustreError: 19430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 319.595764] LustreError: 19391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788358835 with bad export cookie 5595489207568066410 [ 319.599684] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.600451] LustreError: 19391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 319.722572] Lustre: server umount lustre-MDT0001 complete [ 323.752972] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 325.105837] Key type lgssc unregistered [ 325.273205] LNet: 22962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.280568] LNetError: 22962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 325.289596] LNet: Removed LNI 192.168.204.102@tcp [ 325.684161] Key type .llcrypt unregistered [ 325.685777] Key type ._llcrypt unregistered [ 336.053288] Key type ._llcrypt registered [ 336.059297] Key type .llcrypt registered [ 336.116529] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 336.578472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 336.588540] alg: No test for adler32 (adler32-zlib) [ 337.453579] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 337.561885] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 339.167224] Key type lgssc registered [ 339.673789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 343.146673] loop: module loaded [ 343.165366] loop0: detected capacity change from 0 to 204800000 [ 343.238429] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.243392] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.374507] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 345.366143] loop0: detected capacity change from 0 to 204800000 [ 345.429038] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 345.433596] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 345.437819] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 347.327242] loop0: detected capacity change from 0 to 204800000 [ 349.406093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 349.412541] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.499523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 350.517503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 350.552591] Lustre: lustre-MDT0000: new disk, initializing [ 350.578459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 350.585581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 352.100302] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 355.735090] loop1: detected capacity change from 0 to 204800000 [ 357.865393] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 357.992416] Lustre: lustre-OST07c6: new disk, initializing [ 357.995073] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 357.997388] Lustre: Skipped 1 previous similar message [ 357.999856] Lustre: 26130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 358.030359] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 360.046688] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 367.612908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 367.615507] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 367.625637] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 372.675100] Lustre: 26138:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 372.677982] Lustre: 26138:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 372.680324] Lustre: 26138:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 372.683059] Lustre: 26138:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 372.687101] Lustre: 26138:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 372.690552] Lustre: 26138:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 384.227259] Lustre: 25113:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 384.233577] Lustre: 25113:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 384.238362] Lustre: 25113:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 384.242166] Lustre: 25113:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 384.247984] Lustre: 25113:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 384.252243] Lustre: 25113:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 387.726863] Lustre: 25114:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 387.730918] Lustre: 25114:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 387.732775] Lustre: 25114:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 387.734732] Lustre: 25114:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.737329] Lustre: 25114:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 387.739641] Lustre: 25114:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.743592] Lustre: 25114:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 387.745757] Lustre: 25114:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.747624] Lustre: 25114:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 387.749732] Lustre: 25114:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.752297] Lustre: 25114:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 387.754556] Lustre: 25114:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.804094] Lustre: 25113:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 390.807270] Lustre: 25113:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 390.810150] Lustre: 25113:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 390.812759] Lustre: 25113:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.815602] Lustre: 25113:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 390.817689] Lustre: 25113:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.819774] Lustre: 25113:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 390.822115] Lustre: 25113:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.824863] Lustre: 25113:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 390.826975] Lustre: 25113:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.829581] Lustre: 25113:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 390.831569] Lustre: 25113:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 397.393342] Lustre: 26138:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 397.396929] Lustre: 26138:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 397.398877] Lustre: 26138:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 397.400759] Lustre: 26138:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 397.402732] Lustre: 26138:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 397.405683] Lustre: 26138:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 397.408429] Lustre: 26138:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 397.411405] Lustre: 26138:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 397.414125] Lustre: 26138:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 397.416904] Lustre: 26138:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 397.419027] Lustre: 26138:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 397.421186] Lustre: 26138:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 407.473227] Lustre: 25113:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 407.477555] Lustre: 25113:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 407.480615] Lustre: 25113:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 407.483595] Lustre: 25113:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 407.487298] Lustre: 25113:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 407.491278] Lustre: 25113:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 407.495176] Lustre: 25113:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 407.499097] Lustre: 25113:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 407.503185] Lustre: 25113:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 407.506705] Lustre: 25113:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 407.510119] Lustre: 25113:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 407.513126] Lustre: 25113:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 454.638927] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 454.645684] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.660127] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 459.744743] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 464.351409] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 464.471873] Lustre: server umount lustre-OST07c6 complete [ 473.928828] Lustre: server umount lustre-MDT0000 complete [ 494.570753] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 497.587648] Key type lgssc unregistered [ 497.897729] LNet: 28728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.915942] LNetError: 28728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 497.952916] LNet: Removed LNI 192.168.204.102@tcp [ 498.799843] Key type .llcrypt unregistered [ 498.803800] Key type ._llcrypt unregistered [ 517.180227] Key type ._llcrypt registered [ 517.182163] Key type .llcrypt registered [ 517.314553] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 519.278192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 519.372297] alg: No test for adler32 (adler32-zlib) [ 520.717776] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 521.227061] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 522.919771] Key type lgssc registered [ 524.511756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 534.097385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 534.128620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.830378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.949377] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 545.256484] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.287869] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 550.370211] LustreError: 30002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.507874] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.611271] LustreError: 30002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.638670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.176263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 569.436636] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 578.736309] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 585.906873] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 592.802383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 593.088142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.821797] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 605.385831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 607.407504] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 622.052215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 622.058191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.063546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 622.080192] Lustre: Skipped 1 previous similar message [ 622.099053] Lustre: Skipped 1 previous similar message [ 627.176971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 627.181910] Lustre: Skipped 1 previous similar message [ 627.398683] Lustre: server umount lustre-OST0000 complete [ 632.295152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.298894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.315180] Lustre: Skipped 1 previous similar message [ 632.329067] Lustre: Skipped 1 previous similar message [ 637.324992] Lustre: server umount lustre-MDT0000 complete [ 637.408808] LustreError: 29998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.426855] LustreError: 29998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 640.977441] LustreError: 29983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359156 with bad export cookie 10224946274932154441 [ 640.986675] LustreError: 29983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 640.987292] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.354884] Lustre: server umount lustre-MDT0001 complete [ 651.660786] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 654.194860] Key type lgssc unregistered [ 654.461762] LNet: 33737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.471169] LNetError: 33737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.487324] LNet: Removed LNI 192.168.204.102@tcp [ 655.292198] Key type .llcrypt unregistered [ 655.295457] Key type ._llcrypt unregistered [ 671.477175] Key type ._llcrypt registered [ 671.479304] Key type .llcrypt registered [ 671.588390] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 672.259668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.286617] alg: No test for adler32 (adler32-zlib) [ 673.372026] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 673.553685] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 675.201312] Key type lgssc registered [ 676.126744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 682.602403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 682.615800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 683.994269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 688.504504] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 694.243975] LustreError: 34987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.258099] LustreError: 34987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 696.331526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.692457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 700.930592] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 707.669847] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 713.248831] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 720.072708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 720.546492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.782967] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 729.854021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 732.511701] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 745.441977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 745.449574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.463798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 745.485845] Lustre: Skipped 1 previous similar message [ 745.502506] Lustre: Skipped 1 previous similar message [ 749.747852] Lustre: server umount lustre-OST0000 complete [ 755.682382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.683190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.705987] Lustre: Skipped 1 previous similar message [ 755.723063] Lustre: Skipped 1 previous similar message [ 760.061744] Lustre: server umount lustre-MDT0000 complete [ 760.804341] LustreError: 37276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.830204] LustreError: 37276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 763.947848] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359279 with bad export cookie 5248303913086433958 [ 763.955418] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 763.962815] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 764.301208] Lustre: server umount lustre-MDT0001 complete [ 774.171946] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 776.607444] Key type lgssc unregistered [ 776.853621] LNet: 38518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.862223] LNetError: 38518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.887850] LNet: Removed LNI 192.168.204.102@tcp [ 777.572601] Key type .llcrypt unregistered [ 777.577868] Key type ._llcrypt unregistered [ 797.207320] Key type ._llcrypt registered [ 797.209400] Key type .llcrypt registered [ 797.289407] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 798.048770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 798.093307] alg: No test for adler32 (adler32-zlib) [ 799.176653] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 799.413606] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 801.055362] Key type lgssc registered [ 802.464477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 808.501689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 808.525726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.007811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 813.729348] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 820.192801] LustreError: 39915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.222941] LustreError: 39915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 822.921247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.431065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 828.415771] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 835.051500] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 839.546822] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 844.090183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 844.322322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.989682] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 852.951217] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 854.576156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 864.225439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 864.233984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.241487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 864.739448] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.756530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 864.761739] Lustre: Skipped 2 previous similar messages [ 868.778697] Lustre: server umount lustre-MDT0000 complete [ 869.762580] LustreError: 39911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.837276] LustreError: 39895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359386 with bad export cookie 10807233414525777828 [ 870.841145] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.843971] LustreError: 39895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 874.883621] LustreError: 40918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 874.899303] LustreError: 40918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 874.905781] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.2@tcp (stopping) [ 874.978690] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.991905] Lustre: Skipped 1 previous similar message [ 880.001888] LustreError: 39911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 880.002053] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.2@tcp (stopping) [ 880.015269] LustreError: 39911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 880.025323] Lustre: Skipped 2 previous similar messages [ 885.121375] LustreError: 39910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.121395] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.2@tcp (stopping) [ 885.133869] LustreError: 39910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 885.148389] Lustre: Skipped 1 previous similar message [ 885.215579] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 885.396412] Lustre: server umount lustre-MDT0001 complete [ 898.352937] Lustre: server umount lustre-OST0000 complete [ 906.189469] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 907.510877] Key type lgssc unregistered [ 907.654422] LNet: 43775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.659187] LNetError: 43775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.676795] LNet: Removed LNI 192.168.204.102@tcp [ 908.148296] Key type .llcrypt unregistered [ 908.150583] Key type ._llcrypt unregistered [ 921.112658] Key type ._llcrypt registered [ 921.114329] Key type .llcrypt registered [ 921.183315] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 921.773403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 921.791407] alg: No test for adler32 (adler32-zlib) [ 922.755198] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 922.928887] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 924.543550] Key type lgssc registered [ 925.188254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 929.401339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 929.418125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.683247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.045597] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 938.091303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.398636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 940.883685] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 945.572907] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 949.010932] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 953.358454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 953.557781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.628608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 956.821348] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 961.202286] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 971.233702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 971.240794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.251573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.223382] Lustre: lustre-OST0000: Not available for connect from 192.168.204.2@tcp (stopping) [ 974.308800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 974.313367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.429401] Lustre: server umount lustre-OST0000 complete [ 991.357235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.372625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.378592] Lustre: Skipped 3 previous similar messages [ 994.464942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.471495] Lustre: Skipped 1 previous similar message [ 997.617809] Lustre: server umount lustre-MDT0000 complete [ 999.797437] LustreError: 45904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359515 with bad export cookie 10697201158073645428 [ 999.801449] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.807272] LustreError: 45904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1014.239839] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1014.360201] Lustre: server umount lustre-MDT0001 complete [ 1019.575208] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1020.903694] Key type lgssc unregistered [ 1021.067867] LNet: 48878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.073280] LNetError: 48878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.083370] LNet: Removed LNI 192.168.204.102@tcp [ 1021.522246] Key type .llcrypt unregistered [ 1021.525160] Key type ._llcrypt unregistered [ 1035.103934] Key type ._llcrypt registered [ 1035.105989] Key type .llcrypt registered [ 1035.170151] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1035.746649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.762497] alg: No test for adler32 (adler32-zlib) [ 1036.658323] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1036.773017] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1038.383340] Key type lgssc registered [ 1038.887415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1042.445337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1042.455249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.726108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.870983] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1050.705408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.943860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1053.498460] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1057.915260] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1060.893827] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1064.648376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1064.855254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.152733] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1068.973885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1072.687182] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1081.158413] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1083.163673] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1094.315208] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0210h33m29s [ 1095.106147] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1096.673239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.677710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.683465] Lustre: Skipped 2 previous similar messages [ 1096.686223] Lustre: Skipped 1 previous similar message [ 1097.696760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.704789] Lustre: Skipped 3 previous similar messages [ 1100.687428] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.2@tcp (stopping) [ 1102.335431] Lustre: server umount lustre-MDT0000 complete [ 1103.843629] LustreError: 50273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.854354] LustreError: 50273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1103.950502] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1106.821855] LustreError: 52110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.265583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.354687] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.514349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.549206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1108.519920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1110.021192] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1111.839657] Lustre: DEBUG MARKER: Wait for df (36644) ... [ 1112.591501] Lustre: DEBUG MARKER: done [ 1117.153125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.160154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.171692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.175797] Lustre: Skipped 1 previous similar message [ 1117.665247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1120.614666] Lustre: server umount lustre-OST0000 complete [ 1123.809292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.811793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1123.819733] Lustre: Skipped 1 previous similar message [ 1123.823176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.831526] Lustre: Skipped 4 previous similar messages [ 1128.889787] Lustre: server umount lustre-MDT0000 complete [ 1129.956430] LustreError: 51000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.964692] LustreError: 51000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1130.836235] LustreError: 52113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359646 with bad export cookie 6258368252887526166 [ 1130.839907] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.845765] LustreError: 52113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1131.020509] Lustre: server umount lustre-MDT0001 complete [ 1136.196623] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1137.606479] Key type lgssc unregistered [ 1137.781967] LNet: 55187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.787761] LNetError: 55187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.802452] LNet: Removed LNI 192.168.204.102@tcp [ 1138.249321] Key type .llcrypt unregistered [ 1138.251442] Key type ._llcrypt unregistered [ 1159.234131] Key type ._llcrypt registered [ 1159.236152] Key type .llcrypt registered [ 1159.283659] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1159.856637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1159.865806] alg: No test for adler32 (adler32-zlib) [ 1160.762432] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1160.899839] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1162.503162] Key type lgssc registered [ 1163.059628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1167.031784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1167.043299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.207597] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1168.219418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1168.301419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.543118] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1175.572636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.659761] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1175.689085] Lustre: 57374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1175.879518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1178.607625] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1182.845304] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1185.797631] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1189.381244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1189.497767] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1189.623877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.957619] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1196.891816] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1198.139194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1206.635341] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0210h35m21s [ 1207.589407] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1211.025017] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1212.359634] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1212.381515] LustreError: 57379:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1232.798754] Lustre: lustre-MDT0000: Client b0deca4c-98dd-4a89-a2cc-1bb1d3f66947 (at 192.168.204.2@tcp) reconnecting [ 1252.447137] LustreError: 57379:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1253.591079] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1254.888488] Lustre: DEBUG MARKER: done [ 1257.955479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1257.961119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.972938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1259.491230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.494785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1259.506440] Lustre: Skipped 1 previous similar message [ 1263.406685] Lustre: server umount lustre-OST0000 complete [ 1268.194190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1268.198429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.205945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.727097] Lustre: server umount lustre-MDT0000 complete [ 1273.535642] LustreError: 57366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359789 with bad export cookie 10188951839934088724 [ 1273.538677] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.542670] LustreError: 57366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1273.693489] Lustre: server umount lustre-MDT0001 complete [ 1279.037855] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1280.476392] Key type lgssc unregistered [ 1280.649677] LNet: 61697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.653173] LNetError: 61697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.701541] LNet: Removed LNI 192.168.204.102@tcp [ 1282.189293] Key type .llcrypt unregistered [ 1282.192328] Key type ._llcrypt unregistered [ 1304.713655] Key type ._llcrypt registered [ 1304.715247] Key type .llcrypt registered [ 1304.763179] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1305.219574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.234170] alg: No test for adler32 (adler32-zlib) [ 1306.116644] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1306.234380] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1307.839165] Key type lgssc registered [ 1308.312150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1309.864639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1309.879675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.018352] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1311.031196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1311.110113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1311.550492] Lustre: Failing over lustre-MDT0000 [ 1311.667576] Lustre: server umount lustre-MDT0000 complete [ 1314.732082] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:37:10 (1788359830) [ 1317.703236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.983741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.919142] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1323.919250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.997481] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1324.014679] Lustre: 64472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1324.147680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1326.098257] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1329.963816] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1332.717384] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1335.988602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.086837] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1336.192765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.986157] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1339.953443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1342.817833] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1347.417163] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1354.720383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1354.725058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.731216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.235781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.240607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.244333] Lustre: Skipped 1 previous similar message [ 1355.249637] Lustre: Skipped 1 previous similar message [ 1356.598467] Lustre: server umount lustre-MDT0000 complete [ 1358.592601] LustreError: 64465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359874 with bad export cookie 1444767753951044484 [ 1358.596484] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1358.601356] LustreError: 64465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1358.756435] Lustre: server umount lustre-MDT0001 complete [ 1360.371193] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1362.599388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1367.114854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.480046] LustreError: 63396:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bc1c23be x1875231187698048/t0(0) o250->MGC192.168.204.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1368.670665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.726462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1370.747446] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1374.176302] LustreError: 66322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.177428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.189765] LustreError: 66322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1374.860197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.105261] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1377.049276] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1377.119484] Lustre: 63400:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788359876/real 1788359876] req@0000000094936d95 x1875231187697664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788359892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1380.192078] Lustre: 63398:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788359879/real 1788359879] req@0000000021346906 x1875231187697792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788359895 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1380.543386] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1381.343142] Lustre: 63398:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788359881/real 1788359881] req@000000001e049099 x1875231187697920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788359897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1383.380901] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1384.416231] Lustre: 63399:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788359884/real 1788359884] req@0000000009a3e2a2 x1875231187698176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788359900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1390.563870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1390.567676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.573861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.578788] Lustre: Skipped 1 previous similar message [ 1392.777813] Lustre: server umount lustre-MDT0000 complete [ 1394.478583] LustreError: 64464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359910 with bad export cookie 1444767753951046773 [ 1394.483448] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1394.486417] LustreError: 64464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1394.644133] Lustre: server umount lustre-MDT0001 complete [ 1397.743635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1402.209658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.319154] LustreError: 70908:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1404.324553] LustreError: 70908:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000ad886472 x1875231187720064/t0(0) o250->MGC192.168.204.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788359920 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1404.334428] LustreError: 70908:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1404.832563] LustreError: 63396:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000724ec813 x1875231187720832/t0(0) o250->MGC192.168.204.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1404.997951] LustreError: 68989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.015305] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.024526] Lustre: Skipped 2 previous similar messages [ 1405.043602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.048576] Lustre: Skipped 1 previous similar message [ 1405.081576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1406.909914] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1410.530417] LustreError: 68989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.540318] LustreError: 68989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1410.907497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.138785] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1412.143167] Lustre: Skipped 1 previous similar message [ 1412.825681] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1413.151127] Lustre: 63398:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788359912/real 1788359912] req@0000000048df8157 x1875231187720448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788359928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1415.944815] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1418.535477] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1421.280240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1421.283978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.290299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1421.293134] Lustre: Skipped 3 previous similar messages [ 1427.360338] Lustre: server umount lustre-MDT0000 complete [ 1427.428470] LustreError: 70922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.439714] LustreError: 70922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1428.983450] LustreError: 66328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359944 with bad export cookie 1444767753951048166 [ 1428.986700] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1428.989392] LustreError: 66328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1429.115168] Lustre: server umount lustre-MDT0001 complete [ 1431.708471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1433.174250] Lustre: DEBUG MARKER: files compared the same [ 1443.401582] Lustre: server umount lustre-OST0000 complete [ 1450.689670] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1452.061958] Key type lgssc unregistered [ 1452.217560] LNet: 74190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.220314] LNetError: 74190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.231435] LNet: Removed LNI 192.168.204.102@tcp [ 1452.622586] Key type .llcrypt unregistered [ 1452.624231] Key type ._llcrypt unregistered [ 1463.201416] Key type ._llcrypt registered [ 1463.203195] Key type .llcrypt registered [ 1463.257790] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1463.733528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1463.741513] alg: No test for adler32 (adler32-zlib) [ 1464.633787] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1464.749847] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1466.351331] Key type lgssc registered [ 1466.804860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.926462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.937190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.143069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.977501] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug +malloc all [ 1476.942522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.115632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1478.908221] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug +malloc all [ 1482.066176] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1484.631145] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1487.608483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1487.753835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.146437] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug +malloc all [ 1493.038612] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1495.971279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1506.271966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1506.277731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.284204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.296964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.297163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.297762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.309599] Lustre: Skipped 1 previous similar message [ 1508.330626] Lustre: server umount lustre-OST0000 complete [ 1512.226426] Lustre: server umount lustre-MDT0000 complete [ 1512.416074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.416804] LustreError: 75582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.424664] Lustre: Skipped 1 previous similar message [ 1512.432950] LustreError: 75582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1513.880051] LustreError: 76321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360029 with bad export cookie 10575394023438258612 [ 1513.881474] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1513.885399] LustreError: 76321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1514.016344] Lustre: server umount lustre-MDT0001 complete [ 1519.058323] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1520.278473] Key type lgssc unregistered [ 1520.415528] LNet: 79137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.419268] LNetError: 79137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.428315] LNet: Removed LNI 192.168.204.102@tcp [ 1520.836558] Key type .llcrypt unregistered [ 1520.838441] Key type ._llcrypt unregistered [ 1532.302406] Key type ._llcrypt registered [ 1532.304070] Key type .llcrypt registered [ 1532.349179] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1532.791279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1532.863412] alg: No test for adler32 (adler32-zlib) [ 1533.784098] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1533.906859] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1535.503369] Key type lgssc registered [ 1536.116872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1539.850049] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1539.864169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1555.359401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1560.479464] LustreError: 80504:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 1587.103309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.449126] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1592.372320] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1595.304788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.376967] Lustre: *** cfs_fail_loc=706, val=0*** [ 1595.537383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.565539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1597.118694] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1600.394800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.550046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1602.141711] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1604.804582] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1606.921297] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1610.721126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1610.725342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.731875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1611.232145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1611.235783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.242363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1616.865486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1616.868682] Lustre: Skipped 2 previous similar messages [ 1621.983214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1621.986971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1621.990568] Lustre: Skipped 1 previous similar message [ 1622.042279] Lustre: server umount lustre-OST0000 complete [ 1626.080314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1626.083919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.091438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.792088] Lustre: server umount lustre-MDT0000 complete [ 1631.308186] LustreError: 80510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360147 with bad export cookie 7420345698859116609 [ 1631.311450] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1631.314079] LustreError: 80510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1631.436094] Lustre: server umount lustre-MDT0001 complete [ 1635.562914] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1636.679743] Key type lgssc unregistered [ 1636.807407] LNet: 84116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.810689] LNetError: 84116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.819530] LNet: Removed LNI 192.168.204.102@tcp [ 1637.159085] Key type .llcrypt unregistered [ 1637.160724] Key type ._llcrypt unregistered [ 1648.446573] Key type ._llcrypt registered [ 1648.448616] Key type .llcrypt registered [ 1648.497717] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1648.948528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.968730] alg: No test for adler32 (adler32-zlib) [ 1649.851976] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1649.963717] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1651.568172] Key type lgssc registered [ 1652.062431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1655.625616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1655.637308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.363597] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1661.265041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.464687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1663.050347] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1666.527951] LustreError: 86195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1666.535875] LustreError: 86195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1666.698409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1666.856988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.185158] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1671.970028] LustreError: 86199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1671.978950] LustreError: 86199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1672.943076] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.951242] Lustre: MGS: Client 3318829d-7f24-4bd3-b942-d22ad4d39311 (at 0@lo) reconnecting [ 1673.070143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.106087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1674.812307] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1685.472274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1685.476197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.482632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1688.547788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1688.549976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1688.551508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.635371] Lustre: server umount lustre-OST0000 complete [ 1693.665869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.666562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1693.672836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.678173] Lustre: Skipped 2 previous similar messages [ 1697.292837] Lustre: server umount lustre-MDT0000 complete [ 1698.784439] LustreError: 86198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.803965] LustreError: 86198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1699.150641] Lustre: server umount lustre-MDT0001 complete [ 1700.843078] Lustre: server umount MGS complete [ 1706.102385] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:43:41 (1788360221) [ 1717.578484] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 1721.526700] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1736.864190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.676326] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1743.498819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.602394] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1743.629679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1743.670261] Lustre: lustre-MDT0001: new disk, initializing [ 1743.709123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1743.720992] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1743.724050] Lustre: 93354:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1743.728282] LustreError: 93354:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1745.412385] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1750.742293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1750.905255] Lustre: lustre-OST0000: new disk, initializing [ 1750.908198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1750.912619] Lustre: 94121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1750.950119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.590752] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1756.704885] Lustre: 94606:0:(client.c:1642:after_reply()) @@@ resending request on EINPROGRESS req@000000001a9ecd67 x1875231548410112/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788360288 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1758.239442] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1761.076914] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.084191] Lustre: MGS: Client 2bbf65eb-fcb6-42a2-96b3-d6db7c60de6f (at 0@lo) reconnecting [ 1761.106209] Lustre: 92624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1761.116050] LustreError: 93340:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1761.133473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1761.176354] Lustre: lustre-MDT0000: new disk, initializing [ 1761.245953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.258030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1762.271754] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1762.277263] Lustre: 95191:0:(client.c:1642:after_reply()) @@@ resending request on EINPROGRESS req@00000000f7073fb0 x1875231548416128/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788360294 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1762.914897] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1764.389457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1764.393897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1764.410705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1767.334510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1767.341963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1789.921562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.924042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1789.927574] Lustre: Skipped 2 previous similar messages [ 1789.933355] Lustre: Skipped 1 previous similar message [ 1793.759248] Lustre: server umount lustre-OST0000 complete [ 1800.160830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1800.160954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.161715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.173937] Lustre: Skipped 1 previous similar message [ 1801.615817] Lustre: server umount lustre-MDT0000 complete [ 1809.658952] Lustre: server umount lustre-MDT0001 complete [ 1811.733576] Lustre: server umount MGS complete [ 1817.961923] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:45:33 (1788360333) [ 1826.645922] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1827.956961] Key type lgssc unregistered [ 1828.122426] LNet: 97685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.125591] LNetError: 97685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.135734] LNet: Removed LNI 192.168.204.102@tcp [ 1828.539450] Key type .llcrypt unregistered [ 1828.541915] Key type ._llcrypt unregistered [ 1837.814043] Key type ._llcrypt registered [ 1837.816151] Key type .llcrypt registered [ 1837.868895] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1838.292233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.364884] alg: No test for adler32 (adler32-zlib) [ 1839.286269] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1839.403505] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1841.007298] Key type lgssc registered [ 1841.508178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1845.128272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1845.141951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.396485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.432473] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1852.589534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.770407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1854.730725] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1858.202795] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1861.009105] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1864.039830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1864.188395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.480277] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1869.598128] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1871.336738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 1886.689980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1886.696795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.706272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1887.716638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1887.716814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1887.724335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.516651] Lustre: server umount lustre-OST0000 complete [ 1892.833626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.835063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.842510] Lustre: Skipped 1 previous similar message [ 1892.845389] Lustre: Skipped 1 previous similar message [ 1897.441965] Lustre: server umount lustre-MDT0000 complete [ 1897.952049] LustreError: 98883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.958520] LustreError: 98883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1899.019994] LustreError: 98865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360414 with bad export cookie 9750566615547828316 [ 1899.021831] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.027740] LustreError: 98865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1899.177458] Lustre: server umount lustre-MDT0001 complete [ 1903.769668] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1905.053100] Key type lgssc unregistered [ 1905.209548] LNet: 102435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.214520] LNetError: 102435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.225570] LNet: Removed LNI 192.168.204.102@tcp [ 1905.604438] Key type .llcrypt unregistered [ 1905.606090] Key type ._llcrypt unregistered [ 1914.124169] Key type ._llcrypt registered [ 1914.125767] Key type .llcrypt registered [ 1914.173683] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1914.646584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1914.654956] alg: No test for adler32 (adler32-zlib) [ 1915.548552] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 1915.656935] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1917.263357] Key type lgssc registered [ 1917.764647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1920.396737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1920.400838] LustreError: 103498:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1920.401911] Lustre: Skipped 1 previous similar message [ 1920.408593] LustreError: 103497:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1920.412676] LustreError: 103498:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1920.416878] LustreError: 103498:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1920.423223] LustreError: 103498:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1920.424399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.638058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.914877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.046758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1926.852051] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1928.783617] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing lsmod [ 1929.872319] LustreError: 104702:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1929.872397] LustreError: 104703:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1929.879142] LustreError: 104702:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1929.883030] LustreError: 104702:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1929.888193] LustreError: 104702:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1929.888585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1930.042581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.057150] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.057796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.063350] Lustre: Skipped 1 previous similar message [ 1931.067275] Lustre: Skipped 2 previous similar messages [ 1935.203200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.206390] Lustre: Skipped 2 previous similar messages [ 1937.188282] Lustre: server umount lustre-MDT0000 complete [ 1938.815449] LustreError: 103515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360454 with bad export cookie 10567458373282234536 [ 1938.818569] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1938.820327] LustreError: 103515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1939.255226] LustreError: 103966:0:(client.c:1384:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e64b1d51 x1875231827312000/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1939.328691] Lustre: server umount lustre-MDT0001 complete [ 1951.161088] Lustre: server umount lustre-OST0000 complete [ 1954.615832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.798142] LustreError: 105767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1954.807626] LustreError: 105767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1954.833564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1956.406325] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1959.554962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.685399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1961.088960] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1963.631650] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1965.591185] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1968.065159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1968.177804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.321465] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1973.054817] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1979.433892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 1989.601487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1989.603205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.603780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1989.614257] Lustre: Skipped 2 previous similar messages [ 1991.712369] Lustre: server umount lustre-OST0000 complete [ 1994.721757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.722632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1994.728075] Lustre: Skipped 1 previous similar message [ 1994.729586] Lustre: Skipped 1 previous similar message [ 1999.399875] Lustre: server umount lustre-MDT0000 complete [ 1999.840962] LustreError: 105768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.848501] LustreError: 105768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2001.195210] LustreError: 105748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360516 with bad export cookie 10567458373282235208 [ 2001.197849] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.204092] LustreError: 105748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2001.333769] Lustre: server umount lustre-MDT0001 complete [ 2005.765839] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2006.907828] Key type lgssc unregistered [ 2007.041439] LNet: 109319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.045838] LNetError: 109319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.057327] LNet: Removed LNI 192.168.204.102@tcp [ 2007.384119] Key type .llcrypt unregistered [ 2007.385927] Key type ._llcrypt unregistered [ 2017.180290] Key type ._llcrypt registered [ 2017.181584] Key type .llcrypt registered [ 2017.227180] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2017.668161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2017.675060] alg: No test for adler32 (adler32-zlib) [ 2018.538509] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 2018.630966] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2020.215123] Key type lgssc registered [ 2020.635350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2023.723973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2023.733066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.903468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.471105] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2029.886044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.016926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2031.523727] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2034.230216] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2036.320041] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2038.830350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2038.966203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.115689] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2043.807672] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2050.214879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 2056.564342] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2057.385425] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2060.771074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2060.771776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.773223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2060.774354] LustreError: Skipped 1 previous similar message [ 2060.779778] Lustre: Skipped 1 previous similar message [ 2064.542394] Lustre: server umount lustre-OST0000 complete [ 2071.009319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.012038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.012592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.012597] Lustre: Skipped 1 previous similar message [ 2071.027316] Lustre: Skipped 1 previous similar message [ 2072.295718] Lustre: server umount lustre-MDT0000 complete [ 2074.861761] LustreError: 110691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360590 with bad export cookie 10643369923214125484 [ 2074.863638] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2074.880993] LustreError: 110691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2075.176851] Lustre: server umount lustre-MDT0001 complete [ 2086.699169] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2089.787734] Key type lgssc unregistered [ 2090.207304] LNet: 114368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.227680] LNetError: 114368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.243951] LNet: Removed LNI 192.168.204.102@tcp [ 2091.341137] Key type .llcrypt unregistered [ 2091.347533] Key type ._llcrypt unregistered [ 2109.534073] Key type ._llcrypt registered [ 2109.537407] Key type .llcrypt registered [ 2109.658930] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2111.361031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2111.489967] alg: No test for adler32 (adler32-zlib) [ 2112.599392] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 2113.015664] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2114.775260] Key type lgssc registered [ 2116.363715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2125.672503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2125.708237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.385947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.511118] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2137.570957] LustreError: 115594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.585145] LustreError: 115594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2138.842937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.102109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2142.959180] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2148.989926] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2153.780670] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.490622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.872296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.849928] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2172.005425] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2173.163869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2183.650555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2183.659227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.661933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2183.665859] Lustre: Skipped 1 previous similar message [ 2183.671425] Lustre: Skipped 1 previous similar message [ 2185.771373] Lustre: server umount lustre-OST0000 complete [ 2188.769213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.770965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2188.776338] Lustre: Skipped 1 previous similar message [ 2188.789511] Lustre: Skipped 1 previous similar message [ 2193.889194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.902183] Lustre: Skipped 1 previous similar message [ 2194.269572] Lustre: server umount lustre-MDT0000 complete [ 2196.826477] LustreError: 118091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360712 with bad export cookie 11205558295688882136 [ 2196.829240] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.836952] LustreError: 118091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2197.068152] Lustre: server umount lustre-MDT0001 complete [ 2204.146463] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2206.217282] Key type lgssc unregistered [ 2206.468333] LNet: 119128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.477146] LNetError: 119128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.495742] LNet: Removed LNI 192.168.204.102@tcp [ 2207.165495] Key type .llcrypt unregistered [ 2207.167561] Key type ._llcrypt unregistered [ 2224.576791] Key type ._llcrypt registered [ 2224.578852] Key type .llcrypt registered [ 2224.662645] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2225.341446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2225.358869] alg: No test for adler32 (adler32-zlib) [ 2226.332400] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 2226.520275] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2228.143172] Key type lgssc registered [ 2228.772035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2233.636429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2233.654910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.922269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.911866] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2243.603464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.886181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2247.397356] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2252.641173] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2256.447337] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2260.634177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.856214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.335761] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2269.416925] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2279.287574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 2285.113338] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2287.845483] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2287.850963] Lustre: Skipped 2 previous similar messages [ 2290.817666] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2290.826118] Lustre: Skipped 2 previous similar messages [ 2292.769426] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2292.792990] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2296.114612] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.2@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2296.123611] Lustre: Skipped 2 previous similar messages [ 2303.028777] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.2@tcp 192.168.0.[2,10]@tcp [ 2306.458780] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.2/24@tcp in log lustre-MDT0000 [ 2306.465930] Lustre: Skipped 2 previous similar messages [ 2309.374876] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.2/24@tcp [ 2309.383106] Lustre: Skipped 1 previous similar message [ 2315.232544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2315.241347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.256338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2315.265443] Lustre: Skipped 1 previous similar message [ 2316.147324] Lustre: server umount lustre-OST0000 complete [ 2320.353713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.363386] Lustre: Skipped 1 previous similar message [ 2320.368491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2320.373276] Lustre: Skipped 1 previous similar message [ 2324.975575] Lustre: server umount lustre-MDT0000 complete [ 2325.473032] LustreError: 120522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.488419] LustreError: 120522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2327.653178] LustreError: 123175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360843 with bad export cookie 10499121814139825817 [ 2327.654909] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.659574] LustreError: 123175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2327.851169] Lustre: server umount lustre-MDT0001 complete [ 2334.712243] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2336.419378] Key type lgssc unregistered [ 2336.601179] LNet: 125426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.609163] LNetError: 125426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.620634] LNet: Removed LNI 192.168.204.102@tcp [ 2337.099335] Key type .llcrypt unregistered [ 2337.102256] Key type ._llcrypt unregistered [ 2356.111202] Key type ._llcrypt registered [ 2356.117547] Key type .llcrypt registered [ 2356.244859] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2357.089963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2357.108242] alg: No test for adler32 (adler32-zlib) [ 2358.120704] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 2358.346776] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2360.031393] Key type lgssc registered [ 2360.949359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2366.672921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2366.693707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.013484] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2368.022615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.040236] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.2/24@tcp [ 2371.265416] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2377.814102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.053639] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2378.076714] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.2/24@tcp [ 2378.106479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2381.499435] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2387.220905] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.115420] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2396.099502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.356627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.524875] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2407.759902] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2413.747496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 2425.835858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2425.843904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.858712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2429.410509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2429.412307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2429.413116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.415252] Lustre: Skipped 1 previous similar message [ 2430.580175] Lustre: server umount lustre-OST0000 complete [ 2434.529107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.538648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.543019] Lustre: Skipped 1 previous similar message [ 2439.654512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.659983] Lustre: Skipped 1 previous similar message [ 2444.768719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2448.351150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2448.522324] Lustre: server umount lustre-MDT0000 complete [ 2449.898283] LustreError: 126966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.907865] LustreError: 126966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2451.193801] LustreError: 126947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360966 with bad export cookie 10677127158947190887 [ 2451.195620] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2451.204615] LustreError: 126947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2451.421876] Lustre: server umount lustre-MDT0001 complete [ 2458.583910] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2460.542634] Key type lgssc unregistered [ 2460.795575] LNet: 130575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2460.807099] LNetError: 130575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2460.820621] LNet: Removed LNI 192.168.204.102@tcp [ 2461.414102] Key type .llcrypt unregistered [ 2461.416117] Key type ._llcrypt unregistered [ 2479.615511] Key type ._llcrypt registered [ 2479.620994] Key type .llcrypt registered [ 2479.704928] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 2490.718062] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2491.393608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.467637] alg: No test for adler32 (adler32-zlib) [ 2492.434156] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 2492.638268] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2494.267845] Key type lgssc registered [ 2495.152821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2520.568733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2520.586974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.758482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2521.802713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2521.853751] Lustre: lustre-MDT0000: new disk, initializing [ 2521.900113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.917205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2524.930627] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2531.999338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.061934] Lustre: 133951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2532.119119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2532.123576] Lustre: Skipped 1 previous similar message [ 2532.196897] Lustre: lustre-MDT0001: new disk, initializing [ 2532.252385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2532.280625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2532.287544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2535.487232] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2542.077464] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2546.136403] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2551.091889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2551.302773] Lustre: lustre-OST0000: new disk, initializing [ 2551.306466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2551.314329] Lustre: 135918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2551.371159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.562545] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2561.577906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2561.583492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2561.655923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2562.456626] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2565.345100] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2565.501279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2568.845132] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2568.983394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2571.749058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2571.760265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.763256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2571.775480] Lustre: Skipped 1 previous similar message [ 2576.389917] Lustre: server umount lustre-OST0000 complete [ 2581.986766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.988067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2581.998826] Lustre: Skipped 1 previous similar message [ 2582.003126] Lustre: Skipped 1 previous similar message [ 2587.105042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.109573] Lustre: Skipped 1 previous similar message [ 2592.226821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.235051] Lustre: Skipped 1 previous similar message [ 2593.247100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2593.387258] Lustre: server umount lustre-MDT0000 complete [ 2596.610369] LustreError: 135917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361112 with bad export cookie 1651926920565560330 [ 2596.613700] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2596.621186] LustreError: 135917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2596.860351] Lustre: server umount lustre-MDT0001 complete [ 2603.058872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.398951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.926397] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2608.611691] LustreError: 138136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.621586] LustreError: 138136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2611.595081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.819268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2614.734464] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2619.816991] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2623.893869] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2628.543498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2628.829193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.812403] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2637.530456] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2640.712786] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2640.834715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2643.519597] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2643.682606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2650.778147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.890138] Lustre: lustre-OST0001: new disk, initializing [ 2650.893257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2650.897391] Lustre: 141504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2650.959689] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2654.621489] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2657.819294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2657.827455] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2657.890753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2659.844280] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2659.958351] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2662.384533] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2662.500049] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2666.028349] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aba915c7000.ost_server_uuid 50 [ 2666.996557] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9aba915c7000.ost_server_uuid in FULL state after 0 sec [ 2673.124693] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.127660] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2673.127731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2673.136797] Lustre: Skipped 1 previous similar message [ 2677.969825] Lustre: server umount lustre-OST0001 complete [ 2678.246724] LustreError: 139954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.258244] LustreError: 139954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2681.312151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2681.316777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.325718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2681.328995] Lustre: Skipped 1 previous similar message [ 2683.360272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.363751] LustreError: 139954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.379184] LustreError: 139954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2686.236953] Lustre: server umount lustre-OST0000 complete [ 2688.480367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2688.482566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.498250] Lustre: Skipped 2 previous similar messages [ 2694.515855] Lustre: server umount lustre-MDT0000 complete [ 2696.446906] LustreError: 141503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361212 with bad export cookie 1651926920565561212 [ 2696.449733] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2696.453756] LustreError: 141503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2696.636870] Lustre: server umount lustre-MDT0001 complete [ 2706.955236] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2708.357529] Key type lgssc unregistered [ 2708.521796] LNet: 144505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.529396] LNetError: 144505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2708.543749] LNet: Removed LNI 192.168.204.102@tcp [ 2708.966731] Key type .llcrypt unregistered [ 2708.969372] Key type ._llcrypt unregistered [ 2728.845168] Key type ._llcrypt registered [ 2728.847067] Key type .llcrypt registered [ 2728.917624] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 2736.999430] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2737.414295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2737.423266] alg: No test for adler32 (adler32-zlib) [ 2738.315415] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 2738.446662] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2740.055288] Key type lgssc registered [ 2740.747248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2759.431882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2759.445922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.574593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2760.589225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2760.631896] Lustre: lustre-MDT0000: new disk, initializing [ 2760.663364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.673733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2762.694354] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2768.745299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.804946] Lustre: 148663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2768.837933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2768.840168] Lustre: Skipped 1 previous similar message [ 2768.887772] Lustre: lustre-MDT0001: new disk, initializing [ 2768.943190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2768.967324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2768.975989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2771.113675] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2776.594292] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2779.337797] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2782.844398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.986553] Lustre: lustre-OST0000: new disk, initializing [ 2782.989615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2782.994943] Lustre: 150630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2783.050663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.338164] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2786.585227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2786.590557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2786.688450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2791.554214] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2794.007295] Lustre: Failing over lustre-OST0000 [ 2794.099863] Lustre: server umount lustre-OST0000 complete [ 2808.906084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2809.121392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.126649] Lustre: Skipped 1 previous similar message [ 2809.136306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.152667] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2812.473921] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2812.895202] Lustre: 146877:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361312/real 1788361312] req@0000000000a5d95f x1875232690303488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788361328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2813.347180] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2814.471127] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2814.473243] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2814.653589] Lustre: Failing over lustre-MDT0000 [ 2814.810958] Lustre: server umount lustre-MDT0000 complete [ 2814.945391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2814.949862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.015246] Lustre: 146877:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361317/real 1788361317] req@00000000cfcf112a x1875232690304640/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788361333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2818.033155] Lustre: 146877:0:(client.c:2493:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2819.554337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.559983] Lustre: Skipped 1 previous similar message [ 2823.552305] LustreError: 148670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.564884] LustreError: 148670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2823.647523] Lustre: 146874:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361323/real 1788361323] req@0000000096ce8532 x1875232690305792/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788361339 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2824.673919] LustreError: 150085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.683761] LustreError: 150085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2828.672220] LustreError: 151347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.790443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.870296] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.047714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.098723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2831.267362] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2832.322326] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2835.432291] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2835.435066] Lustre: Skipped 1 previous similar message [ 2835.445434] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2835.471176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2840.033568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2840.040371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.050570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2843.444204] Lustre: server umount lustre-OST0000 complete [ 2845.664972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.665055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2845.665829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2845.665834] Lustre: Skipped 1 previous similar message [ 2845.675056] Lustre: Skipped 2 previous similar messages [ 2845.681738] LustreError: Skipped 1 previous similar message [ 2850.785702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.795705] Lustre: Skipped 2 previous similar messages [ 2851.660925] Lustre: server umount lustre-MDT0000 complete [ 2853.689596] LustreError: 149466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361369 with bad export cookie 17123453517080693533 [ 2853.692719] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.694546] LustreError: 149466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2853.860676] Lustre: server umount lustre-MDT0001 complete [ 2859.323266] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2860.626121] Key type lgssc unregistered [ 2860.762694] LNet: 154277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2860.767552] LNetError: 154277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.780547] LNet: Removed LNI 192.168.204.102@tcp [ 2861.199154] Key type .llcrypt unregistered [ 2861.200813] Key type ._llcrypt unregistered [ 2885.066767] Key type ._llcrypt registered [ 2885.068785] Key type .llcrypt registered [ 2885.128361] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2885.665970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2885.676320] alg: No test for adler32 (adler32-zlib) [ 2886.576963] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 2886.695405] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2888.295221] Key type lgssc registered [ 2888.892211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2892.967478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2892.997195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.258655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.348661] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2902.241718] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2904.545272] LustreError: 155794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.554282] LustreError: 155794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2904.946472] LustreError: 155824:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2904.952757] LustreError: 155824:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2904.957748] LustreError: 155824:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 2905.120909] Lustre: server umount lustre-MDT0000 complete [ 2911.978894] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 2916.739658] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2935.930173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.070748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2936.090909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2936.160884] Lustre: lustre-MDT0000: new disk, initializing [ 2936.230633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.240672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2938.446464] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2944.309948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.381432] Lustre: 159120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2944.412904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2944.415920] Lustre: Skipped 1 previous similar message [ 2944.461653] Lustre: lustre-MDT0001: new disk, initializing [ 2944.510765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2944.532130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2944.536016] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2946.311758] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2951.728759] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2954.927895] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2958.525316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2958.683855] Lustre: lustre-OST0000: new disk, initializing [ 2958.686781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2958.694905] Lustre: 161087:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2958.748603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.286577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2959.292828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2959.349508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2961.724493] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2966.970586] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2970.437992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2971.347634] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:04:46 (1788361486) [ 3112.928470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.928688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.937825] Lustre: Skipped 2 previous similar messages [ 3112.946023] Lustre: Skipped 2 previous similar messages [ 3118.048061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.055846] Lustre: Skipped 2 previous similar messages [ 3118.718207] Lustre: server umount lustre-MDT0000 complete [ 3120.779956] LustreError: 159113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361636 with bad export cookie 1442045962215341303 [ 3120.780544] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.788212] LustreError: 159113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3123.169185] LustreError: 159128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3123.170909] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.197450] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3125.215798] LustreError: 161094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.215878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3127.061351] Lustre: server umount lustre-MDT0001 complete [ 3135.191546] Lustre: server umount lustre-OST0000 complete [ 3139.423538] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 3144.321985] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3161.866360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3162.000905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3162.021507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3162.067653] Lustre: lustre-MDT0000: new disk, initializing [ 3162.109120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3162.118645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3164.267204] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3169.779707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.839546] Lustre: 165725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3169.878959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3169.881740] Lustre: Skipped 1 previous similar message [ 3169.926257] Lustre: lustre-MDT0001: new disk, initializing [ 3169.999772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3170.005425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3172.172982] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3177.090390] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3179.721315] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3182.978476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3183.131530] Lustre: lustre-OST0000: new disk, initializing [ 3183.136647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3183.145380] Lustre: 167697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3183.193294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.197469] Lustre: Skipped 1 previous similar message [ 3186.164860] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3190.824749] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3192.546303] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3193.344405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3193.349458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3193.396138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3193.670122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3195.356336] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3195.450661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3198.435035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3198.435349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.437879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3198.440633] LustreError: Skipped 1 previous similar message [ 3198.447505] Lustre: Skipped 1 previous similar message [ 3202.409300] Lustre: server umount lustre-OST0000 complete [ 3206.113654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3206.118259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.672764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3208.676698] Lustre: Skipped 3 previous similar messages [ 3218.399116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3218.477756] Lustre: server umount lustre-MDT0000 complete [ 3218.912722] LustreError: 166909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.927652] LustreError: 166909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3220.286791] LustreError: 165718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361736 with bad export cookie 1442045962215500133 [ 3220.290737] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3220.294409] LustreError: 165718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3220.431233] Lustre: server umount lustre-MDT0001 complete [ 3224.601735] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:08:59 (1788361739) [ 3231.663311] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 3235.942463] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3251.575276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.705772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3251.725164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3251.777224] Lustre: lustre-MDT0000: new disk, initializing [ 3251.811887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.822051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3253.948489] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3261.083459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.158078] Lustre: 172455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3261.316732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3264.041101] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3270.434375] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3274.243041] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3277.906113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3278.059917] Lustre: 174430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3278.421039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3278.476145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3280.919645] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3286.290513] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3294.180781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.181031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.181802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.181807] Lustre: Skipped 2 previous similar messages [ 3294.186910] Lustre: Skipped 2 previous similar messages [ 3295.801420] Lustre: server umount lustre-OST0000 complete [ 3312.095302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3312.192378] Lustre: server umount lustre-MDT0000 complete [ 3314.072323] LustreError: 172447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361829 with bad export cookie 1442045962215501008 [ 3314.073109] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3314.079319] LustreError: 172447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3314.242336] Lustre: server umount lustre-MDT0001 complete [ 3318.391955] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:10:33 (1788361833) [ 3325.960678] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 3330.826414] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3348.356989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.485320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3348.504173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3348.507512] Lustre: Skipped 3 previous similar messages [ 3348.556554] Lustre: lustre-MDT0000: new disk, initializing [ 3348.559366] Lustre: Skipped 2 previous similar messages [ 3348.605486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.610511] Lustre: Skipped 2 previous similar messages [ 3348.621712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3348.627278] Lustre: Skipped 2 previous similar messages [ 3350.546185] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3356.057911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.127846] Lustre: 178746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3356.261854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3357.979361] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3363.077223] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3365.858062] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3369.012822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.142573] Lustre: 180721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3369.412601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3371.665303] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3376.649373] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3379.680711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3379.685151] LustreError: Skipped 1 previous similar message [ 3379.687460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3379.694312] Lustre: Skipped 2 previous similar messages [ 3379.697188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3379.700211] Lustre: Skipped 8 previous similar messages [ 3385.764932] Lustre: server umount lustre-OST0000 complete [ 3393.648313] Lustre: server umount lustre-MDT0000 complete [ 3395.040195] LustreError: 178756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.049444] LustreError: 178756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3395.293547] LustreError: 178739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361911 with bad export cookie 1442045962215502100 [ 3395.295899] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3395.298867] LustreError: 178739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3395.446830] Lustre: server umount lustre-MDT0001 complete [ 3400.451216] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3401.722899] Key type lgssc unregistered [ 3401.874667] LNet: 182575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3401.879143] LNetError: 182575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3401.895425] LNet: Removed LNI 192.168.204.102@tcp [ 3402.289228] Key type .llcrypt unregistered [ 3402.292369] Key type ._llcrypt unregistered [ 3415.269481] Key type ._llcrypt registered [ 3415.271254] Key type .llcrypt registered [ 3415.320095] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3415.841792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3415.849272] alg: No test for adler32 (adler32-zlib) [ 3416.746910] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 3416.864271] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3418.463234] Key type lgssc registered [ 3418.957790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3422.373766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3422.384686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.595833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.598302] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3429.283720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.467056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3431.180970] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3434.275611] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3436.567451] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3439.421822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.587676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3442.845647] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3446.591734] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3448.810912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3466.208221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3466.211463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.217340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.280480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3469.283175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.283482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.295208] Lustre: Skipped 1 previous similar message [ 3470.433470] Lustre: server umount lustre-OST0000 complete [ 3474.401626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.402342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.408130] Lustre: Skipped 1 previous similar message [ 3474.413980] Lustre: Skipped 1 previous similar message [ 3478.116835] Lustre: server umount lustre-MDT0000 complete [ 3479.520234] LustreError: 185208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.528642] LustreError: 185208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3479.704539] LustreError: 183951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361995 with bad export cookie 1103402538439822875 [ 3479.707822] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3479.710077] LustreError: 183951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3479.845317] Lustre: server umount lustre-MDT0001 complete [ 3484.704346] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3485.981177] Key type lgssc unregistered [ 3486.128523] LNet: 187521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3486.134421] LNetError: 187521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3486.145377] LNet: Removed LNI 192.168.204.102@tcp [ 3486.551309] Key type .llcrypt unregistered [ 3486.552858] Key type ._llcrypt unregistered [ 3498.102783] Key type ._llcrypt registered [ 3498.104153] Key type .llcrypt registered [ 3498.151645] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3498.602539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3498.609770] alg: No test for adler32 (adler32-zlib) [ 3499.483645] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 3499.590757] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3501.191177] Key type lgssc registered [ 3501.651332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3504.815996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3504.826330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.022302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.789108] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3511.281660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.474440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3513.151818] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3515.988821] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3518.401188] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3521.144625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3521.322229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.712086] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3526.811357] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3530.538926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3535.842841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.843676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.854810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3535.859203] Lustre: Skipped 1 previous similar message [ 3538.315647] Lustre: lustre-OST0000: Not available for connect from 192.168.204.2@tcp (stopping) [ 3540.964254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3540.967041] Lustre: Skipped 1 previous similar message [ 3541.679704] Lustre: server umount lustre-OST0000 complete [ 3545.211385] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9aba857ad800.ost_server_uuid 50 [ 3545.932422] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3557.856558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3557.861050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.867024] Lustre: Skipped 1 previous similar message [ 3557.870534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.440867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.788788] Lustre: server umount lustre-MDT0000 complete [ 3564.465743] LustreError: 190755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362080 with bad export cookie 3573907054919915601 [ 3564.468302] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3564.471769] LustreError: 190755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3564.611179] Lustre: server umount lustre-MDT0001 complete [ 3568.195652] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:14:43 (1788362083) [ 3571.103367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.330065] LustreError: 192814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.340056] LustreError: 192814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3571.367980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.243153] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3576.800523] LustreError: 192814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3577.049283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.217217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3578.963643] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3582.009050] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3584.486270] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3587.263292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3587.441856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.772930] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3592.666646] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3595.715987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3595.804964] Lustre: lustre-OST0001: new disk, initializing [ 3595.808159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3595.812391] Lustre: 195651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3596.379895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3596.383704] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3596.438648] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3597.486109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3598.607274] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3603.396135] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3607.010360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3607.012016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.014680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3607.020177] Lustre: Skipped 1 previous similar message [ 3607.022308] Lustre: Skipped 2 previous similar messages [ 3610.636994] Lustre: server umount lustre-OST0000 complete [ 3612.130457] LustreError: 196231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.141653] LustreError: 196231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3613.071416] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3613.155925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3614.774555] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3614.869250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3617.252549] LustreError: 196231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3617.260896] LustreError: 196231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3622.374452] LustreError: 196231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.380980] LustreError: 196231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3627.492154] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3627.493081] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3627.496518] LustreError: Skipped 1 previous similar message [ 3627.502721] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3627.507231] Lustre: Skipped 3 previous similar messages [ 3631.271306] Lustre: server umount lustre-OST0001 complete [ 3633.632072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3637.729191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.734808] Lustre: Skipped 2 previous similar messages [ 3639.019233] Lustre: server umount lustre-MDT0000 complete [ 3640.617457] LustreError: 192796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362156 with bad export cookie 3573907054919916658 [ 3640.619807] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3640.623729] LustreError: 192796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3640.755748] Lustre: server umount lustre-MDT0001 complete [ 3650.475418] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:16:05 (1788362165) [ 3653.049376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.174803] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3653.192103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3653.276049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.278800] Lustre: Skipped 1 previous similar message [ 3654.858218] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3658.169764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.226253] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3658.244070] Lustre: 199065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3659.873819] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3662.622111] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3664.945640] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3667.482759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3667.560427] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3670.098432] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3673.122640] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3674.663082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3675.954323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3676.006615] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3678.374391] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3680.053137] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3681.684239] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3684.832431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3684.832462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.836630] LustreError: Skipped 1 previous similar message [ 3684.842138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3684.846718] Lustre: Skipped 4 previous similar messages [ 3697.119140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3697.158079] Lustre: server umount lustre-OST0000 complete [ 3697.535519] LustreError: 200915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.540970] LustreError: 200915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3710.946613] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3714.079487] Lustre: server umount lustre-OST0001 complete [ 3721.185458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3721.189497] Lustre: Skipped 13 previous similar messages [ 3721.772465] Lustre: server umount lustre-MDT0000 complete [ 3723.422493] LustreError: 199057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362239 with bad export cookie 3573907054919918016 [ 3723.424717] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3723.429798] LustreError: 199057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3723.564592] Lustre: server umount lustre-MDT0001 complete [ 3734.053786] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:17:29 (1788362249) [ 3740.586236] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 3744.250592] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3758.270039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.365775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3758.380851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3758.422918] Lustre: lustre-MDT0000: new disk, initializing [ 3758.451616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.454641] Lustre: Skipped 3 previous similar messages [ 3758.461214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3760.112531] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3764.880987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3764.935240] Lustre: 207124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3764.960276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3764.962665] Lustre: Skipped 1 previous similar message [ 3765.003499] Lustre: lustre-MDT0001: new disk, initializing [ 3765.056178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3765.059766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3766.683875] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3771.080581] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3773.384707] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3776.075655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3776.186395] Lustre: lustre-OST0000: new disk, initializing [ 3776.188619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3776.191802] Lustre: 209092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3778.657348] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3782.660984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3782.664944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3782.703951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3782.910695] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3784.675272] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3784.770560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3786.546980] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3786.634240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3787.745396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3787.745742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3787.749246] LustreError: Skipped 2 previous similar messages [ 3787.754736] Lustre: Skipped 5 previous similar messages [ 3787.757885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3787.761046] Lustre: Skipped 1 previous similar message [ 3789.473266] Lustre: server umount lustre-OST0000 complete [ 3797.098465] Lustre: server umount lustre-MDT0000 complete [ 3797.984432] LustreError: 207136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3797.995032] LustreError: 207136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3798.681455] LustreError: 207119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362314 with bad export cookie 3573907054919919787 [ 3798.683903] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3798.686020] LustreError: 207119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3798.811245] Lustre: server umount lustre-MDT0001 complete [ 3802.064863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.770960] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3806.992382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.728672] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3811.408208] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3813.402402] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3815.928838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3818.573579] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3821.452155] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3823.126918] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3827.312973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3828.926512] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3829.008451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3835.869545] Lustre: server umount lustre-OST0000 complete [ 3838.265547] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3839.872627] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3863.879939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3864.009461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.012636] Lustre: Skipped 5 previous similar messages [ 3866.209623] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3869.059114] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3869.749656] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3872.917401] Lustre: server umount lustre-OST0000 complete [ 3876.510261] Lustre: server umount lustre-MDT0000 complete [ 3877.922385] LustreError: 211332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362393 with bad export cookie 3573907054919920655 [ 3877.926413] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3877.926987] LustreError: 211332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3878.053451] Lustre: server umount lustre-MDT0001 complete [ 3881.562264] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:19:56 (1788362396) [ 3884.447972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.634838] LustreError: 217287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.643453] LustreError: 217287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3886.171704] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3889.508452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.313350] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3894.313426] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3896.517648] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3899.095305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3901.732190] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3904.643151] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3906.271163] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3910.424890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3911.898230] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3911.984917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3914.421300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3914.489645] Lustre: lustre-OST0001: new disk, initializing [ 3914.492083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3914.495681] Lustre: 220681:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3916.694710] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3920.815795] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3922.336705] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3924.481357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3924.485160] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3924.515076] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3925.473489] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3927.044913] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3927.136302] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3929.569253] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3929.570877] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.574316] LustreError: Skipped 2 previous similar messages [ 3929.580854] Lustre: Skipped 4 previous similar messages [ 3929.584499] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3929.587567] Lustre: Skipped 7 previous similar messages [ 3933.953557] Lustre: server umount lustre-OST0001 complete [ 3936.272442] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3936.360366] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3937.917377] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3937.995674] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3950.053798] LustreError: 220033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3950.061071] LustreError: 220033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3961.867525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.983874] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3966.820632] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3966.949022] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3966.952596] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3966.952619] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3967.421202] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3970.235309] Lustre: server umount lustre-OST0001 complete [ 3988.562352] Lustre: server umount lustre-OST0000 complete [ 4004.319144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4004.389886] Lustre: server umount lustre-MDT0000 complete [ 4005.931941] LustreError: 219128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362521 with bad export cookie 3573907054919921586 [ 4005.933195] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.936473] LustreError: 219128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4012.149554] Lustre: server umount lustre-MDT0001 complete [ 4021.054490] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:22:16 (1788362536) [ 4023.427877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.540678] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4023.551349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4023.639709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.642523] Lustre: Skipped 5 previous similar messages [ 4025.146077] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4028.178344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.215341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4028.225707] Lustre: 225935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4029.686954] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4032.289855] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4034.232742] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4036.616013] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4036.692352] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4038.863716] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4041.519392] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4045.225783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4045.247204] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4047.190182] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4049.708619] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4051.230451] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 4052.627966] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4052.706592] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4054.106539] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4054.185237] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4056.123725] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aba889b3000.ost_server_uuid 50 [ 4056.657617] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9aba889b3000.ost_server_uuid in FULL state after 0 sec [ 4057.014332] Lustre: Permanently deactivating lustre-OST0001 [ 4057.021260] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4057.862567] Lustre: Permanently reactivating lustre-OST0001 [ 4061.152613] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4061.156430] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.156605] LustreError: Skipped 6 previous similar messages [ 4061.160559] Lustre: Skipped 6 previous similar messages [ 4065.029463] Lustre: server umount lustre-OST0001 complete [ 4072.530890] Lustre: server umount lustre-OST0000 complete [ 4088.287139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4088.357371] Lustre: server umount lustre-MDT0000 complete [ 4089.694199] LustreError: 227795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362605 with bad export cookie 3573907054919922860 [ 4089.696487] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4089.698156] LustreError: 227795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4089.823339] Lustre: server umount lustre-MDT0001 complete [ 4097.945560] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:23:33 (1788362613) [ 4100.589588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.681595] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4100.692654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4100.697501] Lustre: Skipped 5 previous similar messages [ 4102.151236] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4104.938831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.972200] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4104.983065] Lustre: 232621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4106.436697] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4108.926348] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4110.900662] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4113.231480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4113.307631] Lustre: Permanently deactivating lustre-OST0000 [ 4113.312742] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4115.305344] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4117.808692] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4120.131906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4120.223965] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4120.228078] LustreError: 235497:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 4120.298681] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4122.314902] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4124.767024] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4125.326352] Lustre: Permanently reactivating lustre-OST0000 [ 4127.099933] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4127.102849] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4127.106349] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4127.108379] Lustre: Skipped 1 previous similar message [ 4127.124088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4127.131381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4141.632728] Lustre: server umount lustre-OST0001 complete [ 4146.147760] LustreError: 234466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.155301] LustreError: 234466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 4149.072532] Lustre: server umount lustre-OST0000 complete [ 4156.566353] Lustre: server umount lustre-MDT0000 complete [ 4157.814072] LustreError: 232611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362673 with bad export cookie 3573907054919924491 [ 4157.818121] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4157.819304] LustreError: 232611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4157.929609] Lustre: server umount lustre-MDT0001 complete [ 4161.581148] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 4162.559486] Key type lgssc unregistered [ 4162.694355] LNet: 237472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4162.696409] LNetError: 237472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4162.706280] LNet: Removed LNI 192.168.204.102@tcp [ 4162.997797] Key type .llcrypt unregistered [ 4162.998951] Key type ._llcrypt unregistered [ 4173.705615] Key type ._llcrypt registered [ 4173.707450] Key type .llcrypt registered [ 4173.788499] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 4174.424278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4174.440139] alg: No test for adler32 (adler32-zlib) [ 4175.510434] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 4175.704747] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 4177.359155] Key type lgssc registered [ 4178.227539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4187.246948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4187.258207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.580332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.169269] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4198.752209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4198.843260] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4198.862145] Lustre: 238906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4198.868628] Lustre: Permanently deactivating lustre-MDT0001 [ 4198.877874] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4198.885607] LustreError: 238917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4198.902171] LustreError: 238917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4199.150697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4200.526440] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4200.533685] LustreError: 238947:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4200.547330] LustreError: 238947:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4200.556046] LustreError: 238947:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 1, failed: rc = -108 [ 4203.272751] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4210.229651] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4216.462279] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4225.177943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4225.712238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.043673] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4233.895806] LustreError: 240760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.913989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4236.518236] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4239.332218] LustreError: 241284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.928632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4242.150753] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.218835] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4244.221568] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4244.231529] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4247.560241] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4255.050229] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4257.775368] Lustre: Permanently reactivating lustre-MDT0001 [ 4257.790378] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4257.794906] Lustre: Skipped 2 previous similar messages [ 4263.346166] Lustre: lustre-MDT0001: Client 7b962980-ab43-4ec2-9fc0-a081f2d0d03a (at 192.168.204.2@tcp) reconnecting [ 4267.681796] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4268.552863] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4271.236310] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4271.363248] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4273.644316] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4273.749373] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4275.138104] Lustre: Permanently deactivating lustre-MDT0001 [ 4275.147496] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4275.152694] Lustre: Skipped 2 previous similar messages [ 4275.455355] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4286.432170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4286.436897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.448375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4290.531723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4290.535854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.543931] Lustre: Skipped 2 previous similar messages [ 4292.222650] Lustre: server umount lustre-MDT0000 complete [ 4294.150836] LustreError: 238897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362809 with bad export cookie 14026115638401511641 [ 4294.152441] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.158905] LustreError: 238897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4294.316594] Lustre: server umount lustre-MDT0001 complete [ 4302.349074] Lustre: server umount lustre-OST0000 complete [ 4309.613628] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:27:04 (1788362824) [ 4315.910202] Lustre: server umount lustre-OST0001 complete [ 4319.669523] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 4325.238027] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 4344.252655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.388715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4344.395955] Lustre: Skipped 2 previous similar messages [ 4344.411710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4344.463479] Lustre: lustre-MDT0000: new disk, initializing [ 4344.514774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4344.539445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4346.981530] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4353.376771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.436624] Lustre: 247166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4353.491772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4353.497285] Lustre: Skipped 1 previous similar message [ 4353.529850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4353.533742] Lustre: Skipped 3 previous similar messages [ 4353.556823] Lustre: lustre-MDT0001: new disk, initializing [ 4353.609841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4353.631334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4353.636267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4356.082339] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4361.852561] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4364.987384] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4369.280054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4369.466368] Lustre: lustre-OST0000: new disk, initializing [ 4369.469752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4369.475790] Lustre: 249133:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4373.421975] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4376.596639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4376.612146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4376.694120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4380.109287] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4381.525412] LustreError: 248726:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4387.564471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4387.656236] Lustre: lustre-OST0001: new disk, initializing [ 4387.658956] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4387.663292] Lustre: 250248:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4387.719928] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4387.724594] Lustre: Skipped 1 previous similar message [ 4391.535229] LustreError: 248726:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4391.551707] LustreError: 249901:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4391.826855] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4395.039104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4395.049993] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4395.111742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4398.537536] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4399.635913] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4401.656227] LustreError: 249901:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4405.220197] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4405.230126] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4405.234412] LustreError: Skipped 1 previous similar message [ 4405.238732] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4405.248347] Lustre: Skipped 1 previous similar message [ 4405.259170] Lustre: Skipped 1 previous similar message [ 4408.976606] Lustre: server umount lustre-OST0001 complete [ 4410.341761] LustreError: 249117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4410.354256] LustreError: 249117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4415.461914] LustreError: 249657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4415.479769] LustreError: 249657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4423.136362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4423.143423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.152598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4423.158745] Lustre: Skipped 1 previous similar message [ 4425.701422] LustreError: 249117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.716709] LustreError: 249117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4429.191270] Lustre: server umount lustre-OST0000 complete [ 4435.424387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4435.429411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.435748] Lustre: Skipped 1 previous similar message [ 4435.440078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4435.445559] Lustre: Skipped 2 previous similar messages [ 4446.175166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4446.305684] Lustre: server umount lustre-MDT0000 complete [ 4448.883749] LustreError: 250247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362964 with bad export cookie 14026115638401514469 [ 4448.885613] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4448.895281] LustreError: 250247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4455.160217] Lustre: server umount lustre-MDT0001 complete [ 4461.740947] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 4463.884743] Key type lgssc unregistered [ 4464.098948] LNet: 252281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4464.106600] LNetError: 252281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4464.128434] LNet: Removed LNI 192.168.204.102@tcp [ 4464.814444] Key type .llcrypt unregistered [ 4464.820022] Key type ._llcrypt unregistered [ 4495.668676] Key type ._llcrypt registered [ 4495.672476] Key type .llcrypt registered [ 4495.770186] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 4496.465955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4496.479291] alg: No test for adler32 (adler32-zlib) [ 4497.440091] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 4497.626815] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 4499.271163] Key type lgssc registered [ 4500.011688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4505.710592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4505.725221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.941602] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4506.972175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4507.112838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4510.426072] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4516.384564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.448663] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4516.469613] Lustre: 254467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4516.656432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4519.422877] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4524.888567] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4529.152600] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4535.098338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4535.229592] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4535.368738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4541.016158] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4545.150247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4547.179755] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4596.192566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4596.210338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.220760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4596.707682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4596.710762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.737508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4601.831098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4601.842200] Lustre: Skipped 1 previous similar message [ 4602.168406] Lustre: server umount lustre-OST0000 complete [ 4605.771211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4616.384229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4616.634434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.562691] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4626.854211] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4626.861898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4626.865392] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4626.894879] Lustre: Skipped 1 previous similar message [ 4628.831367] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4667.874629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4667.875961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4667.881152] LustreError: Skipped 1 previous similar message [ 4667.884461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4667.905017] Lustre: Skipped 1 previous similar message [ 4667.926479] Lustre: Skipped 1 previous similar message [ 4672.382484] Lustre: server umount lustre-OST0000 complete [ 4675.552909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4675.559715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.572115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4681.529414] Lustre: server umount lustre-MDT0000 complete [ 4684.371954] LustreError: 254461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363200 with bad export cookie 6277256514747837653 [ 4684.376143] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4684.388092] LustreError: 254461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4684.613034] Lustre: server umount lustre-MDT0001 complete [ 4693.245745] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 4695.569678] Key type lgssc unregistered [ 4695.828601] LNet: 259889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4695.840580] LNetError: 259889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4695.865346] LNet: Removed LNI 192.168.204.102@tcp [ 4696.610107] Key type .llcrypt unregistered [ 4696.612320] Key type ._llcrypt unregistered [ 4714.630758] Key type ._llcrypt registered [ 4714.632856] Key type .llcrypt registered [ 4714.719693] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 4715.461814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4715.478716] alg: No test for adler32 (adler32-zlib) [ 4716.450888] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 4716.655556] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 4718.311176] Key type lgssc registered [ 4719.039248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4724.448807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4724.466770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.808348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4729.225624] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4735.382272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4735.698127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4738.962933] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4744.983325] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4749.532824] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4754.722128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4755.026995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4759.873299] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4765.236345] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4771.435700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4787.172293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4787.178240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4787.188986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4787.192520] Lustre: Skipped 1 previous similar message [ 4792.291706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4792.299260] Lustre: Skipped 1 previous similar message [ 4792.522462] Lustre: server umount lustre-OST0000 complete [ 4797.408681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4797.413283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4797.415713] LustreError: Skipped 1 previous similar message [ 4797.436261] Lustre: Skipped 1 previous similar message [ 4797.443392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4797.451467] Lustre: Skipped 1 previous similar message [ 4801.340306] Lustre: server umount lustre-MDT0000 complete [ 4802.529395] LustreError: 262016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4802.542286] LustreError: 262016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4803.582713] LustreError: 261267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363319 with bad export cookie 9212716437189466013 [ 4803.587267] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4803.594638] LustreError: 261267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4803.765289] Lustre: server umount lustre-MDT0001 complete [ 4811.062979] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 4812.971215] Key type lgssc unregistered [ 4813.168597] LNet: 265594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4813.175534] LNetError: 265594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4813.194546] LNet: Removed LNI 192.168.204.102@tcp [ 4813.735880] Key type .llcrypt unregistered [ 4813.737825] Key type ._llcrypt unregistered [ 4826.186052] Key type ._llcrypt registered [ 4826.192158] Key type .llcrypt registered [ 4826.272928] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 4826.969648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4827.058864] alg: No test for adler32 (adler32-zlib) [ 4828.026466] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 4828.209644] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 4829.855298] Key type lgssc registered [ 4830.711310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4835.941870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4835.966531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4837.251953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.974696] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4845.354447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.604925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4848.221440] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4852.813428] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4856.489971] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4860.588577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4860.818922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4861.869641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4864.505793] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4869.056269] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4881.377074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4881.381224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.387705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4881.392027] Lustre: Skipped 1 previous similar message [ 4886.053461] Lustre: server umount lustre-OST0000 complete [ 4891.618481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.618654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4891.619730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4891.632471] Lustre: Skipped 2 previous similar messages [ 4896.737241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4896.745253] Lustre: Skipped 2 previous similar messages [ 4901.856416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4901.862462] Lustre: Skipped 1 previous similar message [ 4902.368019] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4902.461224] Lustre: server umount lustre-MDT0000 complete [ 4904.804273] LustreError: 268648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363420 with bad export cookie 9488850134101501295 [ 4904.809831] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4904.812080] LustreError: 268648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4905.035537] Lustre: server umount lustre-MDT0001 complete [ 4911.912266] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 4913.739267] Key type lgssc unregistered [ 4913.965675] LNet: 270481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4913.969749] LNetError: 270481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4913.986840] LNet: Removed LNI 192.168.204.102@tcp [ 4914.622665] Key type .llcrypt unregistered [ 4914.624538] Key type ._llcrypt unregistered [ 4931.632579] Key type ._llcrypt registered [ 4931.635273] Key type .llcrypt registered [ 4931.722270] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 4932.448704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4932.514502] alg: No test for adler32 (adler32-zlib) [ 4933.503198] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 4933.667836] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 4935.295164] Key type lgssc registered [ 4936.008582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4941.079829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4941.093202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4942.368569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.294863] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4950.910578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.174608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4954.366903] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4959.630208] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4963.319382] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4967.683158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4967.891460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4968.954789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4971.629256] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4976.440711] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4991.462494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4991.462704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4991.467991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4991.478289] Lustre: Skipped 1 previous similar message [ 4991.488184] Lustre: Skipped 1 previous similar message [ 4996.581301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4996.587256] Lustre: Skipped 1 previous similar message [ 4996.708210] Lustre: server umount lustre-OST0000 complete [ 5001.698553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.700022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5001.713981] Lustre: Skipped 1 previous similar message [ 5001.721607] Lustre: Skipped 1 previous similar message [ 5005.349142] Lustre: server umount lustre-MDT0000 complete [ 5006.816521] LustreError: 273173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.825847] LustreError: 273173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5007.452490] LustreError: 271859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363523 with bad export cookie 6590067186706521739 [ 5007.455322] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5007.459392] LustreError: 271859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5007.643218] Lustre: server umount lustre-MDT0001 complete [ 5013.723750] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 5015.299339] Key type lgssc unregistered [ 5015.472798] LNet: 276179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5015.476673] LNetError: 276179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5015.489098] LNet: Removed LNI 192.168.204.102@tcp [ 5016.008572] Key type .llcrypt unregistered [ 5016.012366] Key type ._llcrypt unregistered [ 5027.647278] Key type ._llcrypt registered [ 5027.649510] Key type .llcrypt registered [ 5027.733217] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5028.308975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5028.323953] alg: No test for adler32 (adler32-zlib) [ 5029.311981] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 5029.488289] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 5031.127560] Key type lgssc registered [ 5031.852579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5037.461084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5037.481885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.793795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.626566] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5047.396987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.647147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5050.349745] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5054.902418] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5058.611928] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5062.963056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5063.209841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.242661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5067.429943] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5072.198334] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5082.597558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5082.599771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5082.600213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5082.606082] Lustre: Skipped 1 previous similar message [ 5085.753542] Lustre: server umount lustre-OST0000 complete [ 5088.743306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5088.747715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.760867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5088.765661] Lustre: Skipped 1 previous similar message [ 5092.832426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5092.832665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.075189] Lustre: server umount lustre-MDT0000 complete [ 5096.483716] LustreError: 277377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363612 with bad export cookie 15914731300373708040 [ 5096.487483] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5096.490815] LustreError: 277377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5096.700506] Lustre: server umount lustre-MDT0001 complete [ 5102.853693] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 5104.466532] Key type lgssc unregistered [ 5104.632724] LNet: 281111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5104.636920] LNetError: 281111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5105.703051] LNet: Removed LNI 192.168.204.102@tcp [ 5106.190330] Key type .llcrypt unregistered [ 5106.192608] Key type ._llcrypt unregistered [ 5112.205763] Key type ._llcrypt registered [ 5112.208089] Key type .llcrypt registered [ 5112.273143] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5121.454408] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 5130.989810] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5131.495325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5131.553401] alg: No test for adler32 (adler32-zlib) [ 5132.464202] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 5132.604248] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 5134.208887] Key type lgssc registered [ 5134.908556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5155.038317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5155.061796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.189136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5156.214299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5156.263882] Lustre: lustre-MDT0000: new disk, initializing [ 5156.304499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.323661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5158.531833] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5164.113626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.171890] Lustre: 284637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5164.206124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5164.209239] Lustre: Skipped 1 previous similar message [ 5164.250921] Lustre: lustre-MDT0001: new disk, initializing [ 5164.311390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5164.332851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5164.337579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5166.344285] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5171.678713] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5174.781154] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5178.445156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5178.582693] Lustre: lustre-OST0000: new disk, initializing [ 5178.588585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5178.593761] Lustre: 286605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5178.652821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5181.856754] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5187.087793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5187.093436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5187.157684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5187.579415] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5189.904349] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5190.017753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5192.080469] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5192.179493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5197.281700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5197.282746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.285293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.288161] LustreError: Skipped 1 previous similar message [ 5197.294374] Lustre: Skipped 1 previous similar message [ 5199.142618] Lustre: server umount lustre-OST0000 complete [ 5202.401227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.402872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5202.416681] Lustre: Skipped 1 previous similar message [ 5202.424119] Lustre: Skipped 2 previous similar messages [ 5207.153983] Lustre: server umount lustre-MDT0000 complete [ 5209.019111] LustreError: 286604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363724 with bad export cookie 15016493698855948300 [ 5209.021893] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.024131] LustreError: 286604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5209.177551] Lustre: server umount lustre-MDT0001 complete [ 5213.305385] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:42:08 (1788363728) [ 5216.302341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.521362] LustreError: 289012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5216.529975] LustreError: 289012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5216.555770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.458626] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5221.857641] LustreError: 289013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5222.508094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.680385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5224.604793] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5228.388329] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5231.472321] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5235.182989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5235.356606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5238.516740] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5242.709675] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5252.063935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5252.070617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5252.081986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5253.096346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5253.930642] Lustre: server umount lustre-OST0000 complete [ 5255.884292] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5255.888823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.896827] Lustre: Skipped 1 previous similar message [ 5255.899944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5255.904100] Lustre: Skipped 2 previous similar messages [ 5256.012459] Lustre: server umount lustre-MDT0000 complete [ 5257.944095] LustreError: 288994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363773 with bad export cookie 15016493698855949182 [ 5257.948486] LustreError: 289751:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5257.949252] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5257.949619] LustreError: 288994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5258.128307] Lustre: server umount lustre-MDT0001 complete [ 5263.545535] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 5264.899091] Key type lgssc unregistered [ 5265.063096] LNet: 292544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5265.068431] LNetError: 292544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5265.076332] LNet: Removed LNI 192.168.204.102@tcp [ 5265.519374] Key type .llcrypt unregistered [ 5265.521406] Key type ._llcrypt unregistered [ 5278.655248] Key type ._llcrypt registered [ 5278.657374] Key type .llcrypt registered [ 5278.717813] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 5286.555649] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5287.057708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5287.083697] alg: No test for adler32 (adler32-zlib) [ 5287.984810] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 5288.110818] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 5289.719753] Key type lgssc registered [ 5290.221663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5312.984367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5312.995749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5314.096939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5314.114659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5314.153918] Lustre: lustre-MDT0000: new disk, initializing [ 5314.182398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5314.190518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5315.851482] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5321.036561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.088184] Lustre: 296344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5321.121289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5321.124731] Lustre: Skipped 1 previous similar message [ 5321.162440] Lustre: lustre-MDT0001: new disk, initializing [ 5321.205261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5321.219422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5321.223176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5322.975676] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5327.701507] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5330.447628] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5333.790041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5333.914561] Lustre: lustre-OST2710: new disk, initializing [ 5333.917418] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5333.923084] Lustre: 298313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5333.970779] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5336.617573] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5341.550291] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5342.726906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5342.736609] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5342.783123] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5344.926233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5345.029185] Lustre: lustre-OST03e8: new disk, initializing [ 5345.032556] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5345.039195] Lustre: 299373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5345.085691] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5347.681351] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5351.126687] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5352.465174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5352.471258] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5352.505611] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5354.719824] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5354.826337] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5356.768376] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5356.884647] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5358.818370] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5358.937475] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5360.904319] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5361.005892] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5367.265166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5367.269690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.276638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.779571] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.784143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.790095] Lustre: Skipped 1 previous similar message [ 5368.895736] Lustre: server umount lustre-MDT0000 complete [ 5370.739696] LustreError: 297140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363886 with bad export cookie 1856441339058227481 [ 5370.744663] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5370.747137] LustreError: 297140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5370.915184] Lustre: server umount lustre-MDT0001 complete [ 5383.122395] Lustre: server umount lustre-OST2710 complete [ 5395.002215] Lustre: server umount lustre-OST03e8 complete [ 5402.994618] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 5407.467236] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5424.268513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.374695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5424.392826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5424.434829] Lustre: lustre-MDT0000: new disk, initializing [ 5424.469332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.478483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5426.277126] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5431.680129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5431.735643] Lustre: 304614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5431.873319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5433.679331] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5438.354576] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5440.907856] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5444.181684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5444.307397] Lustre: lustre-OST0000: new disk, initializing [ 5444.309233] Lustre: Skipped 1 previous similar message [ 5444.311568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5444.314953] Lustre: Skipped 2 previous similar messages [ 5444.318932] Lustre: 306583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5444.361802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.365043] Lustre: Skipped 1 previous similar message [ 5447.174819] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5451.998192] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5453.899331] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5454.337441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5454.341461] Lustre: Skipped 1 previous similar message [ 5454.344441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5454.386803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5455.032560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5457.054576] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5457.160378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5459.427414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5459.434359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.444380] Lustre: Skipped 1 previous similar message [ 5459.449180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5459.454871] Lustre: Skipped 3 previous similar messages [ 5460.194763] Lustre: server umount lustre-OST0000 complete [ 5462.497225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5462.502000] LustreError: Skipped 1 previous similar message [ 5462.504506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5462.511612] Lustre: Skipped 1 previous similar message [ 5462.514409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5468.340287] Lustre: server umount lustre-MDT0000 complete [ 5470.069640] LustreError: 304608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363985 with bad export cookie 1856441339058229147 [ 5470.074437] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5470.076375] LustreError: 304608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5470.223758] Lustre: server umount lustre-MDT0001 complete [ 5474.398491] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:46:29 (1788363989) [ 5475.124338] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5475.939250] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:46:31 (1788363991) [ 5477.302514] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5489.369907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.501186] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5489.514871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5489.585799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5491.304651] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5494.942166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.005650] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5495.019382] Lustre: 310440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5496.928315] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5513.764138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5513.864199] Lustre: Denying initial registration attempt from nid 192.168.204.102@tcp, specified as failover [ 5513.868364] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5513.873315] LustreError: 311894:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5513.877033] LustreError: 311894:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5513.879988] LustreError: 311894:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5513.918852] Lustre: server umount lustre-OST0000 complete [ 5513.921225] LustreError: 311894:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5518.304743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.306569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5518.316863] Lustre: Skipped 2 previous similar messages [ 5518.319454] Lustre: Skipped 2 previous similar messages [ 5528.545142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5528.548138] Lustre: Skipped 4 previous similar messages [ 5531.103133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5531.188854] Lustre: server umount lustre-MDT0000 complete [ 5532.956138] LustreError: 310431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364048 with bad export cookie 1856441339058230008 [ 5532.960536] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5532.961382] LustreError: 310431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5533.110083] Lustre: server umount lustre-MDT0001 complete [ 5538.372333] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 5539.655314] Key type lgssc unregistered [ 5539.803644] LNet: 313015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5539.808297] LNetError: 313015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5539.817965] LNet: Removed LNI 192.168.204.102@tcp [ 5540.236551] Key type .llcrypt unregistered [ 5540.238038] Key type ._llcrypt unregistered [ 5545.790461] Key type ._llcrypt registered [ 5545.791672] Key type .llcrypt registered [ 5545.852830] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5546.249792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5546.278220] alg: No test for adler32 (adler32-zlib) [ 5547.189370] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 5547.304518] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 5548.903296] Key type lgssc registered [ 5549.388968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5577.680221] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5581.392692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5581.402044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.514591] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5582.525382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5582.603507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.321695] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5588.111171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5588.177775] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5588.198851] Lustre: 315752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5588.331098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5590.016322] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5593.175193] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5595.538931] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5611.491214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5611.575198] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5611.645878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5613.908770] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5616.753648] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5617.603634] Lustre: server umount lustre-OST0000 complete [ 5619.168342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5619.171185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.178049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5624.288531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5624.292390] Lustre: Skipped 2 previous similar messages [ 5625.249830] Lustre: server umount lustre-MDT0000 complete [ 5626.566719] LustreError: 315745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364142 with bad export cookie 16857197140954046718 [ 5626.569626] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.571362] LustreError: 315745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5626.696045] Lustre: server umount lustre-MDT0001 complete [ 5630.691866] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 5631.886691] Key type lgssc unregistered [ 5632.039757] LNet: 319315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5632.045625] LNetError: 319315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5632.057440] LNet: Removed LNI 192.168.204.102@tcp [ 5632.417239] Key type .llcrypt unregistered [ 5632.418548] Key type ._llcrypt unregistered [ 5642.486500] Key type ._llcrypt registered [ 5642.487563] Key type .llcrypt registered [ 5642.533338] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5642.866931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5642.874686] alg: No test for adler32 (adler32-zlib) [ 5643.733235] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 5643.834102] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 5645.415162] Key type lgssc registered [ 5645.839709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5648.771500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5648.780284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5649.978147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5651.601392] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5654.898908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.040656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5656.538171] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5659.269680] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5661.500640] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5666.275516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5666.416440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5668.788097] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5671.667748] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5686.240861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5686.243515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5686.245921] Lustre: Skipped 2 previous similar messages [ 5686.249380] Lustre: Skipped 1 previous similar message [ 5688.203213] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.2@tcp (stopping) [ 5691.362297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5691.366564] Lustre: Skipped 2 previous similar messages [ 5692.012118] Lustre: server umount lustre-MDT0000 complete [ 5693.311500] LustreError: 322062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.319387] LustreError: 322062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5693.478310] LustreError: 320691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364209 with bad export cookie 17357845281092415870 [ 5693.482026] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5693.483022] LustreError: 320691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5696.481266] LustreError: 320705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5696.481864] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.493647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5698.431914] LustreError: 320704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.634098] Lustre: server umount lustre-MDT0001 complete [ 5701.942601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5704.962328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5719.007797] LustreError: 320019:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ab67aee9 x1875235736412928/t0(0) o250->MGC192.168.204.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5719.141430] LustreError: 322539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.148619] LustreError: 322539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5719.166498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5719.192323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5720.514463] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5723.296172] LustreError: 324320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.301864] LustreError: 324320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5723.479177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5723.606425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5725.044490] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5727.544668] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5728.740584] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5734.148308] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 5735.966041] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5739.656653] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5744.098896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.100041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5744.100811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5744.100818] Lustre: Skipped 2 previous similar messages [ 5744.104545] Lustre: Skipped 1 previous similar message [ 5754.338848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5754.341471] Lustre: Skipped 4 previous similar messages [ 5755.359134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5755.413841] Lustre: server umount lustre-OST0000 complete [ 5759.456595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5759.457070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.464648] Lustre: Skipped 1 previous similar message [ 5771.231092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5771.292741] Lustre: server umount lustre-MDT0000 complete [ 5772.542329] LustreError: 323511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364288 with bad export cookie 17357845281092423024 [ 5772.547493] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.659548] Lustre: server umount lustre-MDT0001 complete [ 5776.325789] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 5777.422551] Key type lgssc unregistered [ 5777.545329] LNet: 326995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5777.547874] LNetError: 326995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5777.558318] LNet: Removed LNI 192.168.204.102@tcp [ 5777.845417] Key type .llcrypt unregistered [ 5777.846350] Key type ._llcrypt unregistered [ 5787.415837] Key type ._llcrypt registered [ 5787.417390] Key type .llcrypt registered [ 5787.460073] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5787.872351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5787.883241] alg: No test for adler32 (adler32-zlib) [ 5788.753665] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 5788.852994] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 5790.439161] Key type lgssc registered [ 5790.879132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5793.936476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5793.944542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.129474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5796.755224] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5799.939258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.072778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5801.592660] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5804.311545] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5806.411469] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5809.569066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.569826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5809.575768] Lustre: Skipped 1 previous similar message [ 5813.652708] Lustre: server umount lustre-MDT0000 complete [ 5814.752620] LustreError: 328407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5814.759446] LustreError: 328407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5815.036835] LustreError: 329124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364330 with bad export cookie 12842370919628470673 [ 5815.038341] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5815.041298] LustreError: 329124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5815.166486] Lustre: server umount lustre-MDT0001 complete [ 5820.942291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5821.086455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5821.103081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5821.205974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.892074] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5826.275860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.347485] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5826.366351] Lustre: 330976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5826.493434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5828.226750] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5830.890278] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5833.065566] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5838.119609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5838.201897] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5838.206552] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5838.210692] LustreError: 332860:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5838.214702] LustreError: 332860:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5838.218253] LustreError: 332860:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5838.255661] Lustre: server umount lustre-OST0000 complete [ 5838.257572] LustreError: 332860:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5843.875247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5843.974959] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5844.052337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.342310] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5849.042495] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5852.018124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5852.056166] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5852.091067] Lustre: lustre-OST0001: new disk, initializing [ 5852.093763] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5852.097896] Lustre: 334390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5852.154434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5852.168424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5854.608336] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5857.260269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5857.266062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5857.294472] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5857.381049] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5862.370771] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5862.372736] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.375934] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5862.380755] Lustre: Skipped 1 previous similar message [ 5862.384829] Lustre: Skipped 2 previous similar messages [ 5864.637942] Lustre: server umount lustre-OST0001 complete [ 5867.490535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5867.493722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5867.494129] LustreError: 333326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.498289] Lustre: Skipped 1 previous similar message [ 5867.498877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5867.506757] LustreError: Skipped 1 previous similar message [ 5867.522032] Lustre: Skipped 1 previous similar message [ 5872.216733] Lustre: server umount lustre-OST0000 complete [ 5877.728862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5877.728900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.729617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5877.740440] Lustre: Skipped 1 previous similar message [ 5879.903725] Lustre: server umount lustre-MDT0000 complete [ 5881.226338] LustreError: 330967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364396 with bad export cookie 12842370919628471212 [ 5881.228840] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5881.230580] LustreError: 330967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5881.353806] Lustre: server umount lustre-MDT0001 complete [ 5885.228927] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 5886.357846] Key type lgssc unregistered [ 5886.502372] LNet: 336350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5886.505702] LNetError: 336350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5886.516302] LNet: Removed LNI 192.168.204.102@tcp [ 5886.866621] Key type .llcrypt unregistered [ 5886.868409] Key type ._llcrypt unregistered [ 5921.294915] Key type ._llcrypt registered [ 5921.296136] Key type .llcrypt registered [ 5921.344803] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 5927.946897] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 5928.268493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5928.329512] alg: No test for adler32 (adler32-zlib) [ 5929.194792] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 5929.290083] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 5930.871154] Key type lgssc registered [ 5931.333301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5944.478351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5944.485508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.576619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5945.589035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5945.627349] Lustre: lustre-MDT0000: new disk, initializing [ 5945.655489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5945.664440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5947.181272] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5951.614781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.658689] Lustre: 341255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5951.683371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5951.685251] Lustre: Skipped 1 previous similar message [ 5951.719935] Lustre: lustre-MDT0001: new disk, initializing [ 5951.759330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5951.775735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5951.779955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5953.311040] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5957.366089] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5959.341644] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5961.785246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5961.917164] Lustre: lustre-OST0000: new disk, initializing [ 5961.919406] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5961.923807] Lustre: 343220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5961.954964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5964.101865] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 5967.876556] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5969.424335] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5971.966701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5971.971224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5972.011200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5972.566360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5974.213461] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5974.319505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5977.059703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5977.059885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.062951] LustreError: Skipped 1 previous similar message [ 5977.070694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5977.075581] Lustre: Skipped 1 previous similar message [ 5981.205187] Lustre: server umount lustre-OST0000 complete [ 5982.688044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5982.691507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.696526] Lustre: Skipped 1 previous similar message [ 5982.699569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5987.296584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5987.297063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.704759] Lustre: server umount lustre-MDT0000 complete [ 5990.040273] LustreError: 341248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364505 with bad export cookie 12279122794605275343 [ 5990.043115] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5990.046942] LustreError: 341248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5990.178953] Lustre: server umount lustre-MDT0001 complete [ 5993.273239] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:55:08 (1788364508) [ 5995.997571] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:55:11 (1788364511) [ 5999.307449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.509258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.046242] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6004.106310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6004.238649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6005.654438] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6008.274066] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6010.344966] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6012.806189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.931215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6015.019255] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6017.634635] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6031.774755] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6032.394464] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6032.973810] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6033.615419] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6034.335878] Lustre: Failing over lustre-MDT0000 [ 6034.536192] Lustre: server umount lustre-MDT0000 complete [ 6034.912966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6034.915372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.920487] LustreError: 346721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6034.927847] LustreError: 346721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6035.426124] LustreError: 345995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.434396] LustreError: 345995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6038.408162] LustreError: 345991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6040.545125] LustreError: 346013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6040.551028] LustreError: 346013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6045.665914] LustreError: 345995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6045.672721] LustreError: 345995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6047.692083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.739825] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6047.833920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6047.860727] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6048.640151] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6049.322352] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6052.837254] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6052.847642] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6052.864405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6054.956504] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6055.498114] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6057.477571] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6058.202798] Lustre: Failing over lustre-MDT0000 [ 6058.388271] Lustre: server umount lustre-MDT0000 complete [ 6058.892291] LustreError: 347994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.073233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6063.073324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6063.077291] Lustre: Skipped 3 previous similar messages [ 6071.858444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.913550] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6072.083098] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6073.633573] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6074.240214] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6077.412744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6077.415625] Lustre: Skipped 2 previous similar messages [ 6077.426689] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6077.444573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6079.516354] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6080.154917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6087.649178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6087.650759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.652987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.652993] Lustre: Skipped 1 previous similar message [ 6087.661824] Lustre: Skipped 2 previous similar messages [ 6088.738358] Lustre: server umount lustre-MDT0000 complete [ 6090.076349] LustreError: 345974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364605 with bad export cookie 12279122794605278024 [ 6090.079532] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6090.080762] LustreError: 345974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6090.198100] Lustre: server umount lustre-MDT0001 complete [ 6101.855079] Lustre: server umount lustre-OST0000 complete [ 6107.231172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.374872] LustreError: 352342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6107.382137] LustreError: 352342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6107.397049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6107.399849] Lustre: Skipped 1 previous similar message [ 6108.789561] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6111.802663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6113.352173] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6115.884631] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6117.780642] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6120.091366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6122.234315] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6124.759591] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6133.542571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6136.799918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6136.803060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.807116] Lustre: Skipped 1 previous similar message [ 6136.809066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6136.810746] Lustre: Skipped 2 previous similar messages [ 6142.549857] Lustre: server umount lustre-OST0000 complete [ 6149.088391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6149.090256] Lustre: Skipped 5 previous similar messages [ 6158.303126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6158.366731] Lustre: server umount lustre-MDT0000 complete [ 6159.327951] LustreError: 353775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.333613] LustreError: 353775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6159.593172] LustreError: 352324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364675 with bad export cookie 12279122794605278745 [ 6159.596466] LustreError: 352324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6159.596830] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6159.718062] Lustre: server umount lustre-MDT0001 complete [ 6163.191301] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6164.204864] Key type lgssc unregistered [ 6164.336520] LNet: 355892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6164.339556] LNetError: 355892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6164.348434] LNet: Removed LNI 192.168.204.102@tcp [ 6164.672415] Key type .llcrypt unregistered [ 6164.673933] Key type ._llcrypt unregistered [ 6174.227135] Key type ._llcrypt registered [ 6174.228222] Key type .llcrypt registered [ 6174.262074] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 6174.539386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6174.557722] alg: No test for adler32 (adler32-zlib) [ 6175.409034] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 6175.506402] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 6177.095157] Key type lgssc registered [ 6177.524895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6180.406554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6180.414167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6181.553384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6182.879271] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6185.734066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6185.852469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6187.224458] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6189.772626] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6191.723751] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6194.092525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6194.210047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.222229] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6198.622171] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6206.499276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6216.676414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6216.677041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6216.677716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6216.679890] Lustre: Skipped 2 previous similar messages [ 6218.898795] Lustre: server umount lustre-MDT0000 complete [ 6220.095176] LustreError: 357366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364735 with bad export cookie 6074283437170541438 [ 6220.098150] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6220.099387] LustreError: 357366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6220.214429] Lustre: server umount lustre-MDT0001 complete [ 6231.705552] Lustre: server umount lustre-OST0000 complete [ 6236.300415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.455041] LustreError: 361111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6236.462755] LustreError: 361111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6236.479876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6237.937411] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6241.063941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.188290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6242.748845] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6245.446061] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6247.404041] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6249.842075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6249.954806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6252.092854] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6254.717714] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6262.245906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6272.480909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.481897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6272.486301] Lustre: Skipped 1 previous similar message [ 6272.488983] Lustre: Skipped 2 previous similar messages [ 6277.601686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6277.604542] Lustre: Skipped 4 previous similar messages [ 6278.295379] Lustre: server umount lustre-MDT0000 complete [ 6279.668323] LustreError: 361093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364795 with bad export cookie 6074283437170542537 [ 6279.671373] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6279.672942] LustreError: 361093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6279.777796] Lustre: server umount lustre-MDT0001 complete [ 6290.470360] Lustre: server umount lustre-OST0000 complete [ 6295.864142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6296.015764] LustreError: 365041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6296.042501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6297.406672] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6300.395198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6301.912071] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6304.415026] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6306.534989] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6309.158070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6311.518628] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6314.166125] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6320.550698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6330.849743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6330.852262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.852806] LustreError: Skipped 1 previous similar message [ 6330.858016] Lustre: Skipped 1 previous similar message [ 6330.860545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6330.863781] Lustre: Skipped 1 previous similar message [ 6332.046948] Lustre: server umount lustre-OST0000 complete [ 6335.968653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.969767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6335.972526] Lustre: Skipped 1 previous similar message [ 6339.476360] Lustre: server umount lustre-MDT0000 complete [ 6340.731263] LustreError: 365021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364856 with bad export cookie 6074283437170543545 [ 6340.733766] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6340.735710] LustreError: 365021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6340.845718] Lustre: server umount lustre-MDT0001 complete [ 6344.642208] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6345.725518] Key type lgssc unregistered [ 6345.862766] LNet: 368591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6345.866884] LNetError: 368591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6345.875540] LNet: Removed LNI 192.168.204.102@tcp [ 6346.204876] Key type .llcrypt unregistered [ 6346.206367] Key type ._llcrypt unregistered [ 6355.742231] Key type ._llcrypt registered [ 6355.743506] Key type .llcrypt registered [ 6355.782103] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 6356.218044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6356.254684] alg: No test for adler32 (adler32-zlib) [ 6357.117403] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 6357.211192] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 6358.791277] Key type lgssc registered [ 6359.159539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6361.727963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6361.734642] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.737081] LustreError: 370018:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6361.745076] LustreError: 370018:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6361.747624] LustreError: 370018:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6361.750522] LustreError: 370018:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6361.753609] LustreError: 370018:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6364.442644] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6364.445688] LustreError: 370415:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6364.455089] LustreError: 370415:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6364.458024] LustreError: 370415:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6364.461262] LustreError: 370415:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6364.463894] LustreError: 370415:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6370.845195] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6371.869123] Key type lgssc unregistered [ 6371.991316] LNet: 371416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6371.994361] LNetError: 371416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6372.006366] LNet: Removed LNI 192.168.204.102@tcp [ 6372.312509] Key type .llcrypt unregistered [ 6372.313626] Key type ._llcrypt unregistered [ 6378.660422] Key type ._llcrypt registered [ 6378.661587] Key type .llcrypt registered [ 6378.707465] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 6385.132451] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 6385.555211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6385.579461] alg: No test for adler32 (adler32-zlib) [ 6386.446127] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 6386.556513] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 6388.151137] Key type lgssc registered [ 6388.578778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6401.160809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6401.166587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6402.248088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6402.259488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6402.298610] Lustre: lustre-MDT0000: new disk, initializing [ 6402.326157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6402.334233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6404.029046] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6408.799146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6408.850893] Lustre: 374545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6408.882183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6408.886109] Lustre: Skipped 1 previous similar message [ 6408.928815] Lustre: lustre-MDT0001: new disk, initializing [ 6408.975689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6408.995050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6408.999684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6410.533229] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6414.469380] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6416.579408] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6418.859412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6418.965383] Lustre: lustre-OST0000: new disk, initializing [ 6418.968431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6418.972541] Lustre: 376510:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6419.000989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6421.095751] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6424.565730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6424.570137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6424.602695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6424.859918] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6426.277836] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6426.354336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6427.736094] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6427.806263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6429.667284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6429.667572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6429.669591] LustreError: Skipped 1 previous similar message [ 6429.670227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6429.674846] Lustre: Skipped 1 previous similar message [ 6430.541671] Lustre: server umount lustre-OST0000 complete [ 6434.784781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6434.784959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.793265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6434.795338] Lustre: Skipped 2 previous similar messages [ 6437.979571] Lustre: server umount lustre-MDT0000 complete [ 6439.324196] LustreError: 374537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364955 with bad export cookie 8062961181366622732 [ 6439.327243] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6439.327952] LustreError: 374537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6439.447558] Lustre: server umount lustre-MDT0001 complete [ 6442.279331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6442.863837] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:02:38 (1788364958) [ 6445.121055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6445.312912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6446.687923] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6449.508661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.619718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6450.953016] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6453.366194] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6455.341617] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6457.678450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6457.798376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6459.934961] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6462.652439] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6465.213107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6465.287516] Lustre: lustre-OST0001: new disk, initializing [ 6465.290394] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6465.294258] Lustre: 381795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6467.610967] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6471.765169] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6472.710996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6472.715161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6472.745783] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6477.797242] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6477.797971] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6477.799509] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6477.801099] LustreError: Skipped 1 previous similar message [ 6477.809353] Lustre: Skipped 2 previous similar messages [ 6479.873892] Lustre: server umount lustre-OST0001 complete [ 6482.914981] LustreError: 380783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.922370] LustreError: 380783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6488.034630] LustreError: 380782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.041064] LustreError: 380782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6493.155180] LustreError: 380783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6493.155227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.156560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6493.156564] Lustre: Skipped 2 previous similar messages [ 6493.161921] LustreError: 380783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6493.169747] Lustre: Skipped 1 previous similar message [ 6498.207531] Lustre: server umount lustre-OST0000 complete [ 6500.831860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6500.834496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.840085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6500.842047] Lustre: Skipped 1 previous similar message [ 6505.902505] Lustre: server umount lustre-MDT0000 complete [ 6507.528593] LustreError: 381794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365023 with bad export cookie 8062961181366623600 [ 6507.533279] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6507.533779] LustreError: 381794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6513.719587] Lustre: server umount lustre-MDT0001 complete [ 6517.380955] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6518.442723] Key type lgssc unregistered [ 6518.580318] LNet: 383777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6518.583049] LNetError: 383777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6518.591371] LNet: Removed LNI 192.168.204.102@tcp [ 6518.930418] Key type .llcrypt unregistered [ 6518.932051] Key type ._llcrypt unregistered [ 6538.420739] Key type ._llcrypt registered [ 6538.422078] Key type .llcrypt registered [ 6538.465491] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 6538.931503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6538.944669] alg: No test for adler32 (adler32-zlib) [ 6539.818746] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 6539.923811] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 6541.511155] Key type lgssc registered [ 6541.952027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6544.944571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6544.952836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.055714] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6546.065085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6546.133116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.653303] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6550.761606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6550.804269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6550.816412] Lustre: 386552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6550.917311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6552.309403] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6554.909299] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6556.815831] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6558.690131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.691117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6558.695389] Lustre: Skipped 1 previous similar message [ 6563.622393] Lustre: server umount lustre-MDT0000 complete [ 6564.917093] LustreError: 386543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365080 with bad export cookie 9092891443183565544 [ 6564.919582] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6564.921298] LustreError: 386543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6565.031687] Lustre: server umount lustre-MDT0001 complete [ 6568.357168] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:04:43 (1788365083) [ 6570.650843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.823926] LustreError: 389062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6570.829886] LustreError: 389062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6570.844807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6572.305556] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6575.515512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6575.666951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6577.122503] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6579.699668] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6581.652609] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6584.199395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.280711] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6584.357769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6586.453547] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6588.936380] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6598.082611] Lustre: Permanently deactivating lustre-OST0000 [ 6598.088471] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6598.830934] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6599.999616] LustreError: 391901:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6600.003034] LustreError: 391901:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 6600.005848] LustreError: 391901:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6600.612515] LustreError: 391949:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 6600.616047] LustreError: 391949:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6607.378476] Lustre: server umount lustre-OST0000 complete [ 6608.421137] LustreError: 392200:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 6608.424278] LustreError: 392200:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6610.912872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.913819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.917171] Lustre: Skipped 1 previous similar message [ 6610.921228] Lustre: Skipped 2 previous similar messages [ 6616.032248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.034211] Lustre: Skipped 1 previous similar message [ 6619.132647] Lustre: server umount lustre-MDT0000 complete [ 6620.502299] LustreError: 389043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365136 with bad export cookie 9092891443183566111 [ 6620.505582] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6620.507794] LustreError: 389043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6630.940360] Lustre: server umount lustre-MDT0001 complete [ 6634.114436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.589413] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6636.561733] LustreError: 393358:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6638.840419] Lustre: 393697:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.102@tcp [ 6639.576110] Lustre: server umount MGS complete [ 6643.803270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.955148] LustreError: 394420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.960885] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6643.966260] Lustre: Skipped 1 previous similar message [ 6643.988143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6645.539409] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6648.673501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.797550] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6650.347544] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6652.973766] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6655.003501] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6655.701628] Lustre: Permanently reactivating lustre-OST0000 [ 6655.707849] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6655.711078] Lustre: Skipped 2 previous similar messages [ 6676.447141] LustreError: 396024:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6679.359853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6679.483207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6679.485966] Lustre: Skipped 1 previous similar message [ 6681.569469] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6684.162665] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6696.930246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.933271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6696.933312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6696.934885] Lustre: Skipped 1 previous similar message [ 6698.197442] Lustre: server umount lustre-OST0000 complete [ 6699.999876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6700.002405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6701.654397] Lustre: server umount lustre-MDT0000 complete [ 6702.049065] LustreError: 394421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6702.053683] LustreError: 394421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6702.915511] LustreError: 394400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365218 with bad export cookie 9092891443183567133 [ 6702.919101] LustreError: 394400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6702.919309] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6703.038951] Lustre: server umount lustre-MDT0001 complete [ 6707.064568] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6708.215440] Key type lgssc unregistered [ 6708.355571] LNet: 398896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6708.359266] LNetError: 398896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6708.368335] LNet: Removed LNI 192.168.204.102@tcp [ 6708.716628] Key type .llcrypt unregistered [ 6708.717927] Key type ._llcrypt unregistered [ 6715.190333] Key type ._llcrypt registered [ 6715.191690] Key type .llcrypt registered [ 6715.228904] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 6721.675981] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 6722.075737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6722.094080] alg: No test for adler32 (adler32-zlib) [ 6722.956881] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 6723.065154] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 6724.647170] Key type lgssc registered [ 6725.099843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6736.760823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6736.765777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.849338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6737.861295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6737.899472] Lustre: lustre-MDT0000: new disk, initializing [ 6737.932544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.939736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6739.407927] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6743.827078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.875117] Lustre: 402030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6743.901758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6743.904051] Lustre: Skipped 1 previous similar message [ 6743.949442] Lustre: lustre-MDT0001: new disk, initializing [ 6743.991410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6744.006808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6744.010521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6745.571394] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6749.460943] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6751.346783] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6753.767561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.865534] Lustre: lustre-OST0000: new disk, initializing [ 6753.867097] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6753.870279] Lustre: 403995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6753.898666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6755.852420] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6759.642446] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6760.955092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6760.959337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6760.996899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6761.051578] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6761.120610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6762.484943] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6762.557401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6766.048880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6766.050179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.051888] LustreError: Skipped 1 previous similar message [ 6766.057636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.061872] Lustre: Skipped 1 previous similar message [ 6769.295717] Lustre: server umount lustre-OST0000 complete [ 6771.168619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.169718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6771.174260] Lustre: Skipped 2 previous similar messages [ 6776.287931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6776.289456] Lustre: Skipped 2 previous similar messages [ 6776.659025] Lustre: server umount lustre-MDT0000 complete [ 6777.855145] LustreError: 402823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365293 with bad export cookie 15788399562647231640 [ 6777.859165] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6777.859836] LustreError: 402823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6777.980265] Lustre: server umount lustre-MDT0001 complete [ 6780.564061] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6781.052620] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:08:16 (1788365296) [ 6783.334573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.510524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6784.952641] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6787.829604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6787.949643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6789.388751] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6791.891617] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6793.847644] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6796.205659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6796.316191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6798.312364] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6800.780484] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6801.508493] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6812.629473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6817.760738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6817.763871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.769526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6818.792271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6820.311539] Lustre: server umount lustre-OST0000 complete [ 6823.906505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6823.906788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.907468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6823.907474] Lustre: Skipped 2 previous similar messages [ 6823.916208] Lustre: Skipped 2 previous similar messages [ 6827.799180] Lustre: server umount lustre-MDT0000 complete [ 6829.024988] LustreError: 406452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6829.030866] LustreError: 406452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6829.178609] LustreError: 409156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365344 with bad export cookie 15788399562647232522 [ 6829.183865] LustreError: 409156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6829.184128] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6829.325471] Lustre: server umount lustre-MDT0001 complete [ 6833.095198] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6834.114486] Key type lgssc unregistered [ 6834.234410] LNet: 410196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6834.237948] LNetError: 410196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6834.247312] LNet: Removed LNI 192.168.204.102@tcp [ 6834.551100] Key type .llcrypt unregistered [ 6834.552706] Key type ._llcrypt unregistered [ 6842.344345] Key type ._llcrypt registered [ 6842.346162] Key type .llcrypt registered [ 6842.392130] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6842.503260] Key type .llcrypt unregistered [ 6842.505186] Key type ._llcrypt unregistered [ 6849.837947] Key type ._llcrypt registered [ 6849.839618] Key type .llcrypt registered [ 6849.884320] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 6850.207041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6850.227485] alg: No test for adler32 (adler32-zlib) [ 6851.085948] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 6851.180313] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 6852.775192] Key type lgssc registered [ 6853.161750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6856.304951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6856.314245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6857.537055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6859.308063] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6862.743175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6862.860225] LustreError: 412379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.865475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6862.867771] LustreError: 412379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6864.976195] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6867.647731] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6867.936212] LustreError: 412383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.942605] LustreError: 412383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6868.960269] LustreError: 412382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6868.965448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6868.969601] LustreError: 412382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6870.168975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6870.328938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6871.916161] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6880.739776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.745491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6880.745865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6880.750750] Lustre: Skipped 1 previous similar message [ 6885.528380] Lustre: server umount lustre-OST0000 complete [ 6889.953942] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6889.957146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6889.961671] Lustre: Skipped 1 previous similar message [ 6889.963703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6890.976121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6890.976752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.019629] Lustre: server umount lustre-MDT0000 complete [ 6894.321149] LustreError: 412364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365410 with bad export cookie 2667638002113763349 [ 6894.323600] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6894.325043] LustreError: 412364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6894.452948] Lustre: server umount lustre-MDT0001 complete [ 6898.058585] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6899.040689] Key type lgssc unregistered [ 6899.165536] LNet: 415532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6899.167675] LNetError: 415532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6899.174414] LNet: Removed LNI 192.168.204.102@tcp [ 6899.483541] Key type .llcrypt unregistered [ 6899.484709] Key type ._llcrypt unregistered [ 6908.399545] Key type ._llcrypt registered [ 6908.400668] Key type .llcrypt registered [ 6908.433835] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 6908.822490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6908.829348] alg: No test for adler32 (adler32-zlib) [ 6909.672660] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 6909.763762] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 6911.343133] Key type lgssc registered [ 6911.678511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6914.433579] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6914.447226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6929.891190] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6935.007447] LustreError: 416899:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 6961.695244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6966.147754] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6971.078492] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6976.377926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.668125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6976.708595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6979.784485] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6983.138900] LustreError: 417938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6983.152748] LustreError: 417938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6985.727884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.956310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6989.153691] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6994.597315] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6998.565912] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7002.593022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7002.601983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.613089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7006.688929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7006.692821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7006.694203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7006.702053] Lustre: Skipped 1 previous similar message [ 7008.389252] Lustre: server umount lustre-OST0000 complete [ 7011.808956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7011.809795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7011.815191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7011.821706] Lustre: Skipped 1 previous similar message [ 7012.571996] Lustre: server umount lustre-MDT0000 complete [ 7014.707232] LustreError: 416906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365530 with bad export cookie 13783209459023655042 [ 7014.710507] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7014.711980] LustreError: 416906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7014.876328] Lustre: server umount lustre-MDT0001 complete [ 7020.604573] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 7022.223387] Key type lgssc unregistered [ 7022.391108] LNet: 420468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7022.403239] LNetError: 420468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7022.417428] LNet: Removed LNI 192.168.204.102@tcp [ 7022.906445] Key type .llcrypt unregistered [ 7022.908296] Key type ._llcrypt unregistered [ 7036.451292] Key type ._llcrypt registered [ 7036.453163] Key type .llcrypt registered [ 7036.515327] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7037.030736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7037.040068] alg: No test for adler32 (adler32-zlib) [ 7037.951600] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 7038.085195] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 7039.703161] Key type lgssc registered [ 7040.439761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7045.058520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7045.068594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7046.355402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7049.162932] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7054.364223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7054.591469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7057.204160] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7061.722299] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7065.009423] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7068.702859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.938667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.045186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7072.458957] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7076.442282] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7080.419125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7080.427100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7080.437727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.440501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.442205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.453852] Lustre: Skipped 1 previous similar message [ 7083.926477] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.2@tcp (stopping) [ 7083.935076] Lustre: Skipped 2 previous similar messages [ 7086.206859] Lustre: server umount lustre-MDT0000 complete [ 7086.560635] LustreError: 421859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.570460] LustreError: 421859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7087.407208] LustreError: 421844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.2@tcp arrived at 1788365603 with bad export cookie 11956402914860180305 [ 7087.414171] LustreError: 421844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7088.407548] LustreError: 421843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365604 with bad export cookie 11956402914860179409 [ 7088.413569] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7088.413934] LustreError: 421843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7091.680804] LustreError: 422589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7091.681183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7091.690095] LustreError: 422589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7091.698573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7093.728480] LustreError: 421863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7093.739854] LustreError: 421863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7094.578393] Lustre: server umount lustre-OST0000 complete [ 7097.803514] LustreError: 422591:0:(client.c:1384:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e6d6fd57 x1875237198129664/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7097.891985] Lustre: server umount lustre-MDT0001 complete [ 7103.492856] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 7105.138820] Key type lgssc unregistered [ 7105.333062] LNet: 425540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7105.344087] LNetError: 425540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7105.355988] LNet: Removed LNI 192.168.204.102@tcp [ 7105.948826] Key type .llcrypt unregistered [ 7105.951220] Key type ._llcrypt unregistered [ 7121.755911] Key type ._llcrypt registered [ 7121.757903] Key type .llcrypt registered [ 7121.838201] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7122.411429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7122.447225] alg: No test for adler32 (adler32-zlib) [ 7123.359911] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 7123.495974] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 7125.112316] Key type lgssc registered [ 7125.781874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7130.399110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7130.414962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7131.650232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7134.215190] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7139.721105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.955531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7142.330448] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7146.796808] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7150.061430] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7153.806339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7154.030235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7156.965226] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7157.097208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7160.330388] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7162.848763] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7162.853116] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7162.860631] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7166.849902] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.2@tcp (stopping) [ 7167.463116] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7168.811477] Lustre: server umount lustre-MDT0001 complete [ 7172.576452] LustreError: 426935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.578587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.580914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7172.590288] LustreError: 426935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7172.611658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.617465] Lustre: Skipped 2 previous similar messages [ 7177.697664] LustreError: 426936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7177.698613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7177.940859] Lustre: server umount lustre-OST0000 complete [ 7180.331824] Lustre: server umount lustre-MDT0000 complete [ 7187.732699] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 7189.330561] Key type lgssc unregistered [ 7189.506618] LNet: 430615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7189.510583] LNetError: 430615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7189.525967] LNet: Removed LNI 192.168.204.102@tcp [ 7190.037660] Key type .llcrypt unregistered [ 7190.039489] Key type ._llcrypt unregistered [ 7204.781857] Key type ._llcrypt registered [ 7204.784800] Key type .llcrypt registered [ 7204.849559] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 7214.549291] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7215.061071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7215.126054] alg: No test for adler32 (adler32-zlib) [ 7216.115915] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 7216.264189] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 7217.895197] Key type lgssc registered [ 7218.636757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7240.095937] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7248.482529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7248.491154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.602325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7249.618040] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7249.660870] Lustre: lustre-MDT0000: new disk, initializing [ 7249.699268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7249.708505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7251.857600] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7261.002702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7261.168996] Lustre: lustre-OST0000: new disk, initializing [ 7261.173597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7261.178992] Lustre: Skipped 1 previous similar message [ 7261.183035] Lustre: 435623:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7261.233735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7262.820846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7262.827814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7262.846928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7264.855310] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7271.065264] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7282.527664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.614912] Lustre: 434605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7282.665655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7282.733313] Lustre: lustre-MDT0001: new disk, initializing [ 7282.829747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7282.860687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7282.866987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7285.897599] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7292.861941] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7293.015632] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7296.905386] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7297.095681] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7300.578060] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7300.583284] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.591328] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7300.607258] Lustre: Skipped 1 previous similar message [ 7305.700021] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7305.707422] Lustre: Skipped 3 previous similar messages [ 7306.699330] Lustre: server umount lustre-MDT0001 complete [ 7315.940419] LustreError: 434613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.953404] LustreError: 434613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7321.066897] LustreError: 434611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7321.093909] LustreError: 434611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7324.128531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7324.133248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.141884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7324.148354] Lustre: Skipped 1 previous similar message [ 7326.176429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7326.176914] LustreError: 434617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.526299] Lustre: server umount lustre-OST0000 complete [ 7345.120048] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7345.292154] Lustre: server umount lustre-MDT0000 complete [ 7353.112619] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 7354.686996] Key type lgssc unregistered [ 7354.879168] LNet: 439374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7354.883158] LNetError: 439374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7354.893464] LNet: Removed LNI 192.168.204.102@tcp [ 7355.391496] Key type .llcrypt unregistered [ 7355.393143] Key type ._llcrypt unregistered [ 7378.455961] Key type ._llcrypt registered [ 7378.459439] Key type .llcrypt registered [ 7378.526722] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 7378.702538] Key type .llcrypt unregistered [ 7378.705263] Key type ._llcrypt unregistered [ 7389.106663] Key type ._llcrypt registered [ 7389.108698] Key type .llcrypt registered [ 7389.174402] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7389.678633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7389.750498] alg: No test for adler32 (adler32-zlib) [ 7390.720280] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 7390.898121] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 7392.543170] Key type lgssc registered [ 7393.217187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7408.289445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7408.300345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7409.446572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7409.466231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7409.527421] Lustre: lustre-MDT0000: new disk, initializing [ 7409.590335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.604585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7411.939439] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7418.823997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7418.897927] Lustre: 442722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7418.947410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7418.950933] Lustre: Skipped 1 previous similar message [ 7418.999743] Lustre: lustre-MDT0001: new disk, initializing [ 7419.053271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7419.085755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7419.092101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7421.680276] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7427.235536] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7430.424688] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7434.124393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7434.306801] Lustre: lustre-OST0000: new disk, initializing [ 7434.311282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7434.315274] Lustre: 444689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7434.389047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7437.950811] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7443.713520] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7444.499683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7444.506296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7444.563708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7454.691892] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.693207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7454.697498] Lustre: Skipped 1 previous similar message [ 7458.590218] Lustre: server umount lustre-MDT0000 complete [ 7460.978697] LustreError: 444688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365976 with bad export cookie 18024429815989185047 [ 7460.981876] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7460.988213] LustreError: 444688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7461.207985] Lustre: server umount lustre-MDT0001 complete [ 7473.800702] Lustre: server umount lustre-OST0000 complete [ 7479.043785] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:19:54 (1788365994) [ 7483.318649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.560805] LustreError: 446719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7483.578502] LustreError: 446719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7483.606495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.862714] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7488.993427] LustreError: 446720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.533061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.718809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7492.799383] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7496.562726] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7499.439982] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7515.749367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7515.848714] Lustre: Found index 0 for lustre-OST0000, updating log [ 7515.929214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.595924] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7522.261727] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7530.293174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7535.584516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7535.588742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.596165] Lustre: Skipped 1 previous similar message [ 7535.599718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7535.602822] Lustre: Skipped 2 previous similar messages [ 7536.611302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7537.156461] Lustre: server umount lustre-OST0000 complete [ 7550.505954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.506917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7550.515847] Lustre: Skipped 1 previous similar message [ 7556.656074] Lustre: server umount lustre-MDT0000 complete [ 7558.408525] LustreError: 446700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366074 with bad export cookie 18024429815989186440 [ 7558.412498] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7558.417070] LustreError: 446700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7558.571209] Lustre: server umount lustre-MDT0001 complete [ 7562.712190] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:21:17 (1788366077) [ 7576.224833] LNet: 1 local NIs in recovery (showing 1): 192.168.204.102@tcp [ 8173.750526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.056129] LustreError: 453144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.140978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.773604] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8179.169642] LustreError: 453145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8183.273370] LustreError: 453144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8185.253516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.557896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8189.512543] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8197.138768] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8203.025301] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8222.506974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8222.783736] Lustre: Found index 0 for lustre-OST0000, updating log [ 8223.040848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8228.447105] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8229.539041] LNet: 1 local NIs in recovery (showing 1): 192.168.204.102@tcp [ 8235.784808] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8237.672595] Lustre: server umount lustre-OST0000 complete [ 8241.889794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8241.893620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8241.900414] Lustre: Skipped 1 previous similar message [ 8241.904230] Lustre: Skipped 5 previous similar messages [ 8242.978747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8242.984781] Lustre: Skipped 2 previous similar messages [ 8244.004703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8244.011174] Lustre: Skipped 1 previous similar message [ 8246.058848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.073088] Lustre: Skipped 3 previous similar messages [ 8247.068905] Lustre: server umount lustre-MDT0000 complete [ 8247.083601] LustreError: 453873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8247.100823] LustreError: 453873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8250.305178] LustreError: 453126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366766 with bad export cookie 18024429815989187497 [ 8250.318356] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8250.320907] LustreError: 453126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8250.670123] Lustre: server umount lustre-MDT0001 complete [ 8278.526262] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 8286.178644] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8293.024346] LNet: 1 local NIs in recovery (showing 1): 192.168.204.102@tcp [ 8313.775477] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:33:48 (1788366828) [ 8327.103081] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 8329.112555] Key type lgssc unregistered [ 8329.351265] LNet: 459750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8329.358136] LNetError: 459750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8434.860789] LNet: Removed LNI 192.168.204.102@tcp [ 8435.538313] Key type .llcrypt unregistered [ 8435.543332] Key type ._llcrypt unregistered [ 8448.702913] Key type ._llcrypt registered [ 8448.705542] Key type .llcrypt registered [ 8448.768905] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8449.400756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8449.426280] alg: No test for adler32 (adler32-zlib) [ 8450.366706] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 8450.504456] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 8452.135151] Key type lgssc registered [ 8452.986854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8456.255371] LNet: Added LNI 192.168.204.102@tcp100 [8/256/0/180] [ 8456.870990] LNet: Added LNI 192.168.204.102@tcp101 [8/256/0/180] [ 8457.504888] LNet: Added LNI 192.168.204.102@tcp102 [8/256/0/180] [ 8458.170212] LNet: Added LNI 192.168.204.102@tcp103 [8/256/0/180] [ 8458.807829] LNet: Added LNI 192.168.204.102@tcp104 [8/256/0/180] [ 8459.464907] LNet: Added LNI 192.168.204.102@tcp105 [8/256/0/180] [ 8460.176752] LNet: Added LNI 192.168.204.102@tcp106 [8/256/0/180] [ 8460.808837] LNet: Added LNI 192.168.204.102@tcp107 [8/256/0/180] [ 8461.532277] LNet: Added LNI 192.168.204.102@tcp108 [8/256/0/180] [ 8462.279365] LNet: Added LNI 192.168.204.102@tcp109 [8/256/0/180] [ 8463.016528] LNet: Added LNI 192.168.204.102@tcp110 [8/256/0/180] [ 8463.740619] LNet: Added LNI 192.168.204.102@tcp111 [8/256/0/180] [ 8464.515726] LNet: Added LNI 192.168.204.102@tcp112 [8/256/0/180] [ 8465.223490] LNet: Added LNI 192.168.204.102@tcp113 [8/256/0/180] [ 8465.921438] LNet: Added LNI 192.168.204.102@tcp114 [8/256/0/180] [ 8466.616316] LNet: Added LNI 192.168.204.102@tcp115 [8/256/0/180] [ 8467.249946] LNet: Added LNI 192.168.204.102@tcp116 [8/256/0/180] [ 8467.982315] LNet: Added LNI 192.168.204.102@tcp117 [8/256/0/180] [ 8468.759089] LNet: Added LNI 192.168.204.102@tcp118 [8/256/0/180] [ 8469.451322] LNet: Added LNI 192.168.204.102@tcp119 [8/256/0/180] [ 8470.234381] LNet: Added LNI 192.168.204.102@tcp120 [8/256/0/180] [ 8470.961602] LNet: Added LNI 192.168.204.102@tcp121 [8/256/0/180] [ 8471.567904] LNet: Added LNI 192.168.204.102@tcp122 [8/256/0/180] [ 8472.196570] LNet: Added LNI 192.168.204.102@tcp123 [8/256/0/180] [ 8472.822365] LNet: Added LNI 192.168.204.102@tcp124 [8/256/0/180] [ 8473.522608] LNet: Added LNI 192.168.204.102@tcp125 [8/256/0/180] [ 8474.210967] LNet: Added LNI 192.168.204.102@tcp126 [8/256/0/180] [ 8474.876327] LNet: Added LNI 192.168.204.102@tcp127 [8/256/0/180] [ 8475.499486] LNet: Added LNI 192.168.204.102@tcp128 [8/256/0/180] [ 8476.145281] LNet: Added LNI 192.168.204.102@tcp129 [8/256/0/180] [ 8476.751474] LNet: Added LNI 192.168.204.102@tcp130 [8/256/0/180] [ 8477.410072] LNet: Added LNI 192.168.204.102@tcp131 [8/256/0/180] [ 8477.982179] LNet: Added LNI 192.168.204.102@tcp132 [8/256/0/180] [ 8478.613686] LNet: Added LNI 192.168.204.102@tcp133 [8/256/0/180] [ 8479.230748] LNet: Added LNI 192.168.204.102@tcp134 [8/256/0/180] [ 8479.911702] LNet: Added LNI 192.168.204.102@tcp135 [8/256/0/180] [ 8484.770034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8484.794233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8485.867111] Lustre: 462773:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.102@tcp: can't add NID 192.168.204.102@tcp130: rc = -75 [ 8485.909858] LNetError: 460233:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.102 [ 8485.949380] LNetError: Protocol error connecting to 192.168.204.102@tcp116 at host 192.168.204.102:988: is it running a compatible version of Lustre? [ 8485.957870] LNetError: Skipped 4 previous similar messages [ 8486.021634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8486.054335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8486.112554] Lustre: lustre-MDT0000: new disk, initializing [ 8486.185295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8486.208637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8486.368924] LNet: 1 local NIs in recovery (showing 1): 192.168.204.102@tcp126 [ 8488.425631] LNetError: 460236:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.102 [ 8488.431492] LNetError: 460236:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [ 8488.439274] LNetError: Protocol error connecting to 192.168.204.102@tcp126 at host 192.168.204.102:988: is it running a compatible version of Lustre? [ 8488.455160] LNetError: Skipped 7 previous similar messages [ 8489.319990] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8492.513960] LNetError: 462795:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.102 [ 8492.525466] LNetError: Protocol error connecting to 192.168.204.102@tcp126 at host 192.168.204.102:988: is it running a compatible version of Lustre? [ 8497.435593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8497.535336] Lustre: 462788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8497.604544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8497.608036] Lustre: Skipped 1 previous similar message [ 8497.671296] Lustre: lustre-MDT0001: new disk, initializing [ 8497.709463] Lustre: 463594:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.102@tcp: can't add NID 192.168.204.102@tcp130: rc = -75 [ 8497.720570] Lustre: 463594:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8497.796377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8497.831630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8497.841596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8500.706870] LNetError: 462798:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.102 [ 8500.719837] LNetError: Protocol error connecting to 192.168.204.102@tcp126 at host 192.168.204.102:988: is it running a compatible version of Lustre? [ 8500.751635] Lustre: 463824:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.102@tcp: can't add NID 192.168.204.102@tcp130: rc = -75 [ 8500.758951] Lustre: 463824:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8500.932339] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8508.048365] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8512.806352] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8516.065962] LNetError: 462794:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.102 [ 8516.075655] LNetError: Protocol error connecting to 192.168.204.102@tcp128 at host 192.168.204.102:988: is it running a compatible version of Lustre? [ 8516.082772] LNetError: Skipped 1 previous similar message [ 8518.226390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8518.501414] Lustre: lustre-OST0000: new disk, initializing [ 8518.504697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8518.510315] Lustre: 464766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8518.575983] Lustre: 464771:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.102@tcp: can't add NID 192.168.204.102@tcp130: rc = -75 [ 8518.581360] Lustre: 464771:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 8518.615794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8523.358859] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8524.262339] LNetError: 460236:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.102 [ 8524.288679] LNetError: 460236:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 8524.300560] LNetError: Protocol error connecting to 192.168.204.102@tcp109 at host 192.168.204.102:988: is it running a compatible version of Lustre? [ 8524.319548] LNetError: Skipped 6 previous similar messages [ 8528.391997] Lustre: 465400:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.102@tcp: can't add NID 192.168.204.102@tcp130: rc = -75 [ 8528.398713] Lustre: 465400:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8528.448576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8528.454663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8528.563478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8529.979549] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8539.108381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8539.109229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.114609] LustreError: Skipped 1 previous similar message [ 8539.116123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8539.134858] Lustre: Skipped 1 previous similar message [ 8541.666691] LNetError: 462795:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.102 [ 8541.675914] LNetError: 462795:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [ 8541.682116] LNetError: Protocol error connecting to 192.168.204.102@tcp128 at host 192.168.204.102:988: is it running a compatible version of Lustre? [ 8541.689672] LNetError: Skipped 12 previous similar messages [ 8544.228838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8544.237859] Lustre: Skipped 2 previous similar messages [ 8546.783287] LNet: 1 local NIs in recovery (showing 1): 192.168.204.102@tcp126 [ 8549.344604] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8549.347478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8549.360012] Lustre: Skipped 1 previous similar message [ 8549.469905] Lustre: server umount lustre-OST0000 complete [ 8554.465594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8554.466641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8554.474589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8554.480731] Lustre: Skipped 1 previous similar message [ 8559.587421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8559.590904] Lustre: Skipped 2 previous similar messages [ 8566.239128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8566.361838] Lustre: server umount lustre-MDT0000 complete [ 8568.670594] LustreError: 462781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367084 with bad export cookie 18094859896526250265 [ 8568.675672] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8568.678406] LustreError: 462781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8568.848032] Lustre: server umount lustre-MDT0001 complete [ 8576.486228] LNetError: 462798:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.102 [ 8576.501565] LNetError: 462798:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 17 previous similar messages [ 8576.507593] LNetError: Protocol error connecting to 192.168.204.102@tcp117 at host 192.168.204.102:988: is it running a compatible version of Lustre? [ 8576.513824] LNetError: Skipped 17 previous similar messages [ 8579.680842] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 8584.654378] Key type lgssc unregistered [ 8585.524742] LNet: 466680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8585.555582] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8585.585837] LNet: Removed LNI 192.168.204.102@tcp [ 8586.655157] LNet: Removed LNI 192.168.204.102@tcp100 [ 8586.670401] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8586.683803] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8586.698900] LNet: Removed LNI 192.168.204.102@tcp101 [ 8587.743276] LNet: Removed LNI 192.168.204.102@tcp102 [ 8587.747907] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8587.766335] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8587.792507] LNet: Removed LNI 192.168.204.102@tcp103 [ 8588.831352] LNet: Removed LNI 192.168.204.102@tcp104 [ 8589.855251] LNet: Removed LNI 192.168.204.102@tcp105 [ 8589.872621] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8589.887312] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8589.904522] LNet: Removed LNI 192.168.204.102@tcp106 [ 8590.949899] LNet: Removed LNI 192.168.204.102@tcp107 [ 8592.032391] LNet: Removed LNI 192.168.204.102@tcp108 [ 8593.055230] LNet: Removed LNI 192.168.204.102@tcp109 [ 8594.079154] LNet: Removed LNI 192.168.204.102@tcp110 [ 8594.086601] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8594.106789] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8594.121628] LNet: Removed LNI 192.168.204.102@tcp111 [ 8595.167209] LNet: Removed LNI 192.168.204.102@tcp112 [ 8596.191556] LNet: Removed LNI 192.168.204.102@tcp113 [ 8597.215189] LNet: Removed LNI 192.168.204.102@tcp114 [ 8598.241937] LNet: Removed LNI 192.168.204.102@tcp115 [ 8599.264221] LNet: Removed LNI 192.168.204.102@tcp116 [ 8600.287685] LNet: Removed LNI 192.168.204.102@tcp117 [ 8601.375234] LNet: Removed LNI 192.168.204.102@tcp118 [ 8602.402887] LNet: Removed LNI 192.168.204.102@tcp119 [ 8602.406364] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8602.435817] LNetError: 466680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 8602.454947] LNet: Removed LNI 192.168.204.102@tcp120 [ 8603.487224] LNet: Removed LNI 192.168.204.102@tcp121 [ 8604.511178] LNet: Removed LNI 192.168.204.102@tcp122 [ 8605.535165] LNet: Removed LNI 192.168.204.102@tcp123 [ 8606.561131] LNet: Removed LNI 192.168.204.102@tcp124 [ 8607.583724] LNet: Removed LNI 192.168.204.102@tcp125 [ 8607.597263] LNet: Removed LNI 192.168.204.102@tcp126 [ 8608.673436] LNet: Removed LNI 192.168.204.102@tcp127 [ 8609.695293] LNet: Removed LNI 192.168.204.102@tcp128 [ 8610.719318] LNet: Removed LNI 192.168.204.102@tcp129 [ 8610.722076] LNet: Removed LNI 192.168.204.102@tcp130 [ 8610.731811] LNet: Removed LNI 192.168.204.102@tcp131 [ 8610.743990] LNet: Removed LNI 192.168.204.102@tcp132 [ 8610.752549] LNet: Removed LNI 192.168.204.102@tcp133 [ 8610.762574] LNet: Removed LNI 192.168.204.102@tcp134 [ 8610.795708] LNet: Removed LNI 192.168.204.102@tcp135 [ 8611.748176] Key type .llcrypt unregistered [ 8611.753805] Key type ._llcrypt unregistered [ 8640.047526] Key type ._llcrypt registered [ 8640.049170] Key type .llcrypt registered [ 8640.195985] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 8657.199675] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8658.687653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8658.772562] alg: No test for adler32 (adler32-zlib) [ 8659.943724] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 8660.287240] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 8661.999181] Key type lgssc registered [ 8663.140565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8701.686661] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:40:16 (1788367216) [ 8721.273408] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 8724.498164] Key type lgssc unregistered [ 8724.841330] LNet: 470772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8724.862551] LNetError: 470772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8724.874788] LNet: Removed LNI 192.168.204.102@tcp [ 8725.832552] Key type .llcrypt unregistered [ 8725.834529] Key type ._llcrypt unregistered [ 8742.732740] Key type ._llcrypt registered [ 8742.739995] Key type .llcrypt registered [ 8742.843363] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8743.740254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8743.808560] alg: No test for adler32 (adler32-zlib) [ 8744.833724] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 8745.037930] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 8746.703280] Key type lgssc registered [ 8747.798862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8756.940672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8756.993066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8758.293159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8758.331107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8758.428587] Lustre: lustre-MDT0000: new disk, initializing [ 8758.568516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8758.589425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8762.985924] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8773.239389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8773.343895] Lustre: 471983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8773.414256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8773.420084] Lustre: Skipped 1 previous similar message [ 8773.482401] Lustre: lustre-MDT0001: new disk, initializing [ 8773.552407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8773.589170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8773.599667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8778.481167] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8787.596262] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8793.714482] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8800.919963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8801.202683] Lustre: lustre-OST0000: new disk, initializing [ 8801.207155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8801.217259] Lustre: 473955:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8801.290224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8807.820350] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8809.099692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8809.114270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8809.215448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8817.580559] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8821.483747] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing lnet_if_list [ 8825.956965] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8825.991653] LNet: Added LNI 192.168.204.102@tcp10 [8/256/0/180] [ 8832.140927] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8832.172921] LNet: Added LNI 192.168.204.102@tcp11 [8/256/0/180] [ 8836.120419] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8836.147095] LNet: Added LNI 192.168.204.102@tcp12 [8/256/0/180] [ 8839.961614] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8839.982470] LNet: Added LNI 192.168.204.102@tcp13 [8/256/0/180] [ 8844.498586] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8844.530869] LNet: Added LNI 192.168.204.102@tcp14 [8/256/0/180] [ 8849.339420] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8849.396323] LNet: Added LNI 192.168.204.102@tcp15 [8/256/0/180] [ 8854.040269] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8854.108727] LNet: Added LNI 192.168.204.102@tcp16 [8/256/0/180] [ 8859.576350] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8859.607342] LNet: Added LNI 192.168.204.102@tcp17 [8/256/0/180] [ 8864.552698] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8864.595344] LNet: Added LNI 192.168.204.102@tcp18 [8/256/0/180] [ 8870.070817] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8870.132443] LNet: Added LNI 192.168.204.102@tcp19 [8/256/0/180] [ 8876.932611] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8876.993815] LNet: Added LNI 192.168.204.102@tcp20 [8/256/0/180] [ 8883.389933] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8883.430171] LNet: Added LNI 192.168.204.102@tcp21 [8/256/0/180] [ 8889.428401] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8889.473714] LNet: Added LNI 192.168.204.102@tcp22 [8/256/0/180] [ 8896.704065] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8896.752991] LNet: Added LNI 192.168.204.102@tcp23 [8/256/0/180] [ 8903.238311] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8903.274521] LNet: Added LNI 192.168.204.102@tcp24 [8/256/0/180] [ 8909.609236] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8909.643594] LNet: Added LNI 192.168.204.102@tcp25 [8/256/0/180] [ 8915.459301] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8915.497819] LNet: Added LNI 192.168.204.102@tcp26 [8/256/0/180] [ 8921.607260] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8921.647791] LNet: Added LNI 192.168.204.102@tcp27 [8/256/0/180] [ 8926.557322] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8926.586413] LNet: Added LNI 192.168.204.102@tcp28 [8/256/0/180] [ 8931.535333] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8931.561249] LNet: Added LNI 192.168.204.102@tcp29 [8/256/0/180] [ 8936.750365] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8936.788472] LNet: Added LNI 192.168.204.102@tcp30 [8/256/0/180] [ 8942.488400] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8942.523787] LNet: Added LNI 192.168.204.102@tcp31 [8/256/0/180] [ 8948.076683] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8948.109297] LNet: Added LNI 192.168.204.102@tcp32 [8/256/0/180] [ 8953.359450] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8953.387284] LNet: Added LNI 192.168.204.102@tcp33 [8/256/0/180] [ 8958.310366] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8958.335613] LNet: Added LNI 192.168.204.102@tcp34 [8/256/0/180] [ 8963.050926] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8963.072100] LNet: Added LNI 192.168.204.102@tcp35 [8/256/0/180] [ 8967.562340] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8967.581813] LNet: Added LNI 192.168.204.102@tcp36 [8/256/0/180] [ 8972.314836] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8972.333708] LNet: Added LNI 192.168.204.102@tcp37 [8/256/0/180] [ 8976.987801] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8977.010948] LNet: Added LNI 192.168.204.102@tcp38 [8/256/0/180] [ 8982.116362] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8982.150959] LNet: Added LNI 192.168.204.102@tcp39 [8/256/0/180] [ 8987.053743] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8987.077696] LNet: Added LNI 192.168.204.102@tcp40 [8/256/0/180] [ 8992.588270] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8992.627664] LNet: Added LNI 192.168.204.102@tcp41 [8/256/0/180] [ 8998.970911] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8998.997635] LNet: Added LNI 192.168.204.102@tcp42 [8/256/0/180] [ 9005.509226] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9005.544504] LNet: Added LNI 192.168.204.102@tcp43 [8/256/0/180] [ 9011.281934] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9011.306890] LNet: Added LNI 192.168.204.102@tcp44 [8/256/0/180] [ 9017.409893] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9017.435530] LNet: Added LNI 192.168.204.102@tcp45 [8/256/0/180] [ 9023.205637] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9023.234281] LNet: Added LNI 192.168.204.102@tcp46 [8/256/0/180] [ 9028.404097] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9028.424712] LNet: Added LNI 192.168.204.102@tcp47 [8/256/0/180] [ 9033.625316] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9033.649494] LNet: Added LNI 192.168.204.102@tcp48 [8/256/0/180] [ 9039.028238] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9039.053326] LNet: Added LNI 192.168.204.102@tcp49 [8/256/0/180] [ 9044.386426] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9044.406668] LNet: Added LNI 192.168.204.102@tcp50 [8/256/0/180] [ 9059.170890] Lustre: server umount lustre-MDT0000 complete [ 9060.324467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9060.340538] Lustre: Skipped 1 previous similar message [ 9060.347454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9061.687692] LustreError: 471975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367577 with bad export cookie 432665639788419248 [ 9061.692895] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9061.700415] LustreError: 471975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9061.939895] Lustre: server umount lustre-MDT0001 complete [ 9074.739726] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9080.398147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.823434] Lustre: 471257:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367581/real 1788367581] req@00000000c4b47b48 x1875238988165120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9081.841753] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9081.852565] Lustre: Skipped 1 previous similar message [ 9082.591893] Lustre: 471258:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367582/real 1788367582] req@00000000ddc9bfb0 x1875238988165248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9086.943739] Lustre: 471256:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367586/real 1788367586] req@00000000403a8528 x1875238988165376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9087.008792] LustreError: 471254:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003779b6bf x1875238988166272/t0(0) o250->MGC192.168.204.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9087.242478] LustreError: 473949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9087.259389] LustreError: 473949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9087.297912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9090.179413] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9091.362762] LustreError: 498160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9091.372641] LustreError: 498160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9092.384123] Lustre: 471257:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367592/real 1788367592] req@000000003de12c2b x1875238988165760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9092.412802] Lustre: 471257:0:(client.c:2493:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9095.893052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.154961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9097.120848] Lustre: 471257:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367596/real 1788367596] req@00000000e27baa46 x1875238988166016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9099.081211] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9100.267639] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9103.721168] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9107.229452] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9121.761579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9121.771303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9121.785336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9123.296822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9123.302876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9125.860024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9125.866947] Lustre: Skipped 1 previous similar message [ 9130.978844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9130.988085] Lustre: Skipped 1 previous similar message [ 9135.071707] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9135.162748] Lustre: server umount lustre-OST0000 complete [ 9141.218765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9141.225749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9141.234359] Lustre: Skipped 2 previous similar messages [ 9141.243947] Lustre: Skipped 1 previous similar message [ 9143.886686] Lustre: server umount lustre-MDT0000 complete [ 9146.340853] LustreError: 498164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9146.367651] LustreError: 498164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9146.405553] LustreError: 471976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367662 with bad export cookie 432665639788421670 [ 9146.406901] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9146.416755] LustreError: 471976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9146.631526] Lustre: server umount lustre-MDT0001 complete [ 9153.992849] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 9156.045508] Key type lgssc unregistered [ 9156.321643] LNet: 500940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9156.328950] LNetError: 500940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9156.332825] LNet: Removed LNI 192.168.204.102@tcp [ 9156.335209] LNet: Removed LNI 192.168.204.102@tcp10 [ 9156.337933] LNet: Removed LNI 192.168.204.102@tcp11 [ 9156.340811] LNet: Removed LNI 192.168.204.102@tcp12 [ 9156.343265] LNet: Removed LNI 192.168.204.102@tcp13 [ 9156.346352] LNet: Removed LNI 192.168.204.102@tcp14 [ 9156.348948] LNet: Removed LNI 192.168.204.102@tcp15 [ 9156.353136] LNet: Removed LNI 192.168.204.102@tcp16 [ 9156.356196] LNet: Removed LNI 192.168.204.102@tcp17 [ 9156.359112] LNet: Removed LNI 192.168.204.102@tcp18 [ 9156.362785] LNet: Removed LNI 192.168.204.102@tcp19 [ 9156.365246] LNet: Removed LNI 192.168.204.102@tcp20 [ 9156.368637] LNet: Removed LNI 192.168.204.102@tcp21 [ 9156.372529] LNet: Removed LNI 192.168.204.102@tcp22 [ 9156.375217] LNet: Removed LNI 192.168.204.102@tcp23 [ 9156.378342] LNet: Removed LNI 192.168.204.102@tcp24 [ 9156.381652] LNet: Removed LNI 192.168.204.102@tcp25 [ 9156.385450] LNet: Removed LNI 192.168.204.102@tcp26 [ 9156.389325] LNet: Removed LNI 192.168.204.102@tcp27 [ 9156.399181] LNet: Removed LNI 192.168.204.102@tcp28 [ 9156.407305] LNet: Removed LNI 192.168.204.102@tcp29 [ 9156.413803] LNet: Removed LNI 192.168.204.102@tcp30 [ 9156.417342] LNet: Removed LNI 192.168.204.102@tcp31 [ 9156.421044] LNet: Removed LNI 192.168.204.102@tcp32 [ 9156.424663] LNet: Removed LNI 192.168.204.102@tcp33 [ 9156.428505] LNet: Removed LNI 192.168.204.102@tcp34 [ 9156.432215] LNet: Removed LNI 192.168.204.102@tcp35 [ 9156.436172] LNet: Removed LNI 192.168.204.102@tcp36 [ 9156.438936] LNet: Removed LNI 192.168.204.102@tcp37 [ 9156.442081] LNet: Removed LNI 192.168.204.102@tcp38 [ 9156.445289] LNet: Removed LNI 192.168.204.102@tcp39 [ 9156.447300] LNet: Removed LNI 192.168.204.102@tcp40 [ 9156.451124] LNet: Removed LNI 192.168.204.102@tcp41 [ 9156.454756] LNet: Removed LNI 192.168.204.102@tcp42 [ 9156.457938] LNet: Removed LNI 192.168.204.102@tcp43 [ 9156.461522] LNet: Removed LNI 192.168.204.102@tcp44 [ 9156.465127] LNet: Removed LNI 192.168.204.102@tcp45 [ 9156.468519] LNet: Removed LNI 192.168.204.102@tcp46 [ 9156.471668] LNet: Removed LNI 192.168.204.102@tcp47 [ 9156.476128] LNet: Removed LNI 192.168.204.102@tcp48 [ 9156.479979] LNet: Removed LNI 192.168.204.102@tcp49 [ 9156.489123] LNet: Removed LNI 192.168.204.102@tcp50 [ 9157.075177] Key type .llcrypt unregistered [ 9157.079729] Key type ._llcrypt unregistered [ 9171.168445] Key type ._llcrypt registered [ 9171.173244] Key type .llcrypt registered [ 9171.279595] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 9182.440561] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9183.144372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9183.157491] alg: No test for adler32 (adler32-zlib) [ 9184.163449] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 9184.331221] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 9185.967271] Key type lgssc registered [ 9186.843697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9212.401694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9212.423985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9213.613492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9213.643213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9213.726965] Lustre: lustre-MDT0000: new disk, initializing [ 9213.799198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9213.814121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9217.069770] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9225.300663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9225.360654] Lustre: 504117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9225.406207] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9225.410273] Lustre: Skipped 1 previous similar message [ 9225.470080] Lustre: lustre-MDT0001: new disk, initializing [ 9225.553191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9225.584348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9225.595132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9228.758461] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9236.122244] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9240.633717] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9246.488248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9246.645429] Lustre: *** cfs_fail_loc=90f, val=0*** [ 9246.702610] Lustre: lustre-OST0000: new disk, initializing [ 9246.706282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9246.711743] Lustre: 506132:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9246.777067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9251.308403] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9256.984095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9256.995504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9257.062393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9257.908756] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9267.172748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9267.175311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9267.179589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9267.182411] LustreError: Skipped 1 previous similar message [ 9267.187845] Lustre: Skipped 1 previous similar message [ 9268.749120] Lustre: server umount lustre-OST0000 complete [ 9271.777637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9271.788546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9271.800806] Lustre: Skipped 1 previous similar message [ 9271.806947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9277.484155] Lustre: server umount lustre-MDT0000 complete [ 9280.100726] LustreError: 504109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367795 with bad export cookie 11699208667561543778 [ 9280.105156] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9280.108580] LustreError: 504109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9280.387456] Lustre: server umount lustre-MDT0001 complete [ 9287.963535] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 9289.905465] Key type lgssc unregistered [ 9290.094653] LNet: 508034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9290.100883] LNetError: 508034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9291.174061] LNet: Removed LNI 192.168.204.102@tcp [ 9291.704377] Key type .llcrypt unregistered [ 9291.706811] Key type ._llcrypt unregistered [ 9304.166652] Key type ._llcrypt registered [ 9304.168706] Key type .llcrypt registered [ 9304.231445] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9304.810855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9304.874137] alg: No test for adler32 (adler32-zlib) [ 9305.816183] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 9305.977704] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 9307.616108] Key type lgssc registered [ 9308.276507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9312.643741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9312.655668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9313.916248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9316.439562] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9321.719795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.934066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9324.421708] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9329.143666] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9333.064428] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9337.916423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9338.193939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9342.462569] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9347.611960] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9355.810612] loop0: detected capacity change from 0 to 204800000 [ 9355.931455] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9355.937657] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9355.946533] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9360.066764] loop0: detected capacity change from 0 to 204800000 [ 9360.180378] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9360.186761] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9360.196563] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9364.003496] loop0: detected capacity change from 0 to 204800000 [ 9367.671624] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9367.748525] Lustre: fs15246-OST0000: new disk, initializing [ 9367.751658] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9367.757172] Lustre: 512874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9367.796253] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9371.560404] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9378.027824] loop1: detected capacity change from 0 to 204800000 [ 9381.754826] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9381.836685] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9381.876989] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9381.943994] Lustre: fs15246-MDT0000: new disk, initializing [ 9382.050057] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9382.064119] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9382.121339] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9383.137467] Lustre: 513997:0:(client.c:1642:after_reply()) @@@ resending request on EINPROGRESS req@000000000280b031 x1875239576312832/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788367914 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9384.683589] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9388.214349] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9388.223571] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9388.241304] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9403.364030] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9403.373099] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9404.566545] Lustre: server umount fs15246-MDT0000 complete [ 9408.487700] LustreError: 511768:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9408.498551] LustreError: 511768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9410.678741] Lustre: server umount fs15246-OST0000 complete [ 9418.209262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9418.219801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9418.238091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9423.733326] Lustre: server umount lustre-OST0000 complete [ 9428.961854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.963740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9428.970530] Lustre: Skipped 2 previous similar messages [ 9428.975982] Lustre: Skipped 1 previous similar message [ 9432.360488] Lustre: server umount lustre-MDT0000 complete [ 9434.080670] LustreError: 509281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.093053] LustreError: 509281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9434.510876] LustreError: 509262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367950 with bad export cookie 13137629876624605877 [ 9434.514434] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9434.520934] LustreError: 509262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9434.697582] Lustre: server umount lustre-MDT0001 complete [ 9441.337931] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 9443.215214] Key type lgssc unregistered [ 9443.415756] LNet: 517092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9443.425826] LNetError: 517092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9443.440763] LNet: Removed LNI 192.168.204.102@tcp [ 9443.993857] Key type .llcrypt unregistered [ 9443.996214] Key type ._llcrypt unregistered [ 9456.592353] Key type ._llcrypt registered [ 9456.595200] Key type .llcrypt registered [ 9456.666719] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9457.357363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9457.369187] alg: No test for adler32 (adler32-zlib) [ 9458.308262] Lustre: Lustre: Build Version: 2.17.56_1_g134d346 [ 9458.498654] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 9460.127142] Key type lgssc registered [ 9460.834430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9485.810888] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 9492.171790] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9515.781540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9515.796211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9516.956241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9516.982196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9517.030812] Lustre: lustre-MDT0000: new disk, initializing [ 9517.091821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9517.103574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9520.248707] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9527.782167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.862762] Lustre: 521531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9527.910503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9527.912934] Lustre: Skipped 1 previous similar message [ 9527.962068] Lustre: lustre-MDT0001: new disk, initializing [ 9528.020736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9528.054298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9528.064683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9531.111539] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9538.124984] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9542.003846] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9546.550234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9546.699462] Lustre: lustre-OST0000: new disk, initializing [ 9546.702390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9546.707576] Lustre: 523501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9546.764893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.872796] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9555.497643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9555.506268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9555.581908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9557.957259] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9560.729421] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9560.897390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9563.789760] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9563.977220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9565.664161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9565.669183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9565.671142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9565.681620] Lustre: Skipped 1 previous similar message [ 9565.688480] Lustre: Skipped 1 previous similar message [ 9570.789768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9570.795886] Lustre: Skipped 1 previous similar message [ 9571.373723] Lustre: server umount lustre-OST0000 complete [ 9574.375728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9574.385424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9574.401816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9575.907629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9579.881389] Lustre: server umount lustre-MDT0000 complete [ 9582.304135] LustreError: 521525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788368098 with bad export cookie 2189663798153969604 [ 9582.308890] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9582.312739] LustreError: 521525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9582.529566] Lustre: server umount lustre-MDT0001 complete [ 9598.878327] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 9605.194517] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9624.454448] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9466 sec ========== 12:55:39 (1788368139) [ 9625.561500] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:55:40 (1788368140) === [ 9627.753869] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:55:42 (1788368142) === [ 9649.874533] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 9651.373908] Key type lgssc unregistered [ 9651.561469] LNet: 530135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9651.565815] LNetError: 530135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9651.575477] LNet: Removed LNI 192.168.204.102@tcp [ 9652.047716] Key type .llcrypt unregistered [ 9652.049235] Key type ._llcrypt unregistered