[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 926349959 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002428] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004029] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007034] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008027] pid_max: default: 32768 minimum: 301 [ 0.009160] LSM: Security Framework initializing [ 0.010073] Yama: becoming mindful. [ 0.011063] SELinux: Initializing. [ 0.012074] *** VALIDATE selinux *** [ 0.022315] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027194] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028261] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030170] *** VALIDATE tmpfs *** [ 0.032322] *** VALIDATE proc *** [ 0.034075] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.036615] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038190] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040036] Spectre V2 : User space: Vulnerable [ 0.041017] Speculative Store Bypass: Vulnerable [ 0.045066] debug: unmapping init [mem 0xffffffffb9a59000-0xffffffffb9a60fff] [ 0.047300] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048948] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050040] ... version: 2 [ 0.051026] ... bit width: 48 [ 0.052023] ... generic registers: 4 [ 0.053025] ... value mask: 0000ffffffffffff [ 0.054031] ... max period: 00007fffffffffff [ 0.055017] ... fixed-purpose events: 3 [ 0.056030] ... event mask: 000000070000000f [ 0.057462] rcu: Hierarchical SRCU implementation. [ 0.059899] smp: Bringing up secondary CPUs ... [ 0.060856] x86: Booting SMP configuration: [ 0.061039] .... node #0, CPUs: #1 #2 #3 [ 0.073473] smp: Brought up 1 node, 4 CPUs [ 0.075017] smpboot: Max logical packages: 1 [ 0.076104] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.123634] node 0 deferred pages initialised in 45ms [ 0.128202] devtmpfs: initialized [ 0.129332] x86/mm: Memory block size: 128MB [ 0.133248] gcov: version magic: 0x41383552 [ 0.135455] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136200] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137374] pinctrl core: initialized pinctrl subsystem [ 0.138284] [ 0.138932] ************************************************************* [ 0.139023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140021] ** ** [ 0.141032] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142026] ** ** [ 0.143022] ** This means that this kernel is built to expose internal ** [ 0.144022] ** IOMMU data structures, which may compromise security on ** [ 0.145025] ** your system. ** [ 0.146027] ** ** [ 0.147024] ** If you see this message and you are not debugging the ** [ 0.148022] ** kernel, report this immediately to your vendor! ** [ 0.149029] ** ** [ 0.150021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151020] ************************************************************* [ 0.153812] NET: Registered protocol family 16 [ 0.154620] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.155180] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.156235] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.158059] cpuidle: using governor menu [ 0.159000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.163965] PCI: Using configuration type 1 for base access [ 0.167313] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.179189] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.180105] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184072] cryptd: max_cpu_qlen set to 1000 [ 0.187000] ACPI: Added _OSI(Module Device) [ 0.188025] ACPI: Added _OSI(Processor Device) [ 0.190060] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201998] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211598] ACPI: Interpreter enabled [ 0.215144] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218021] ACPI: Using IOAPIC for interrupt routing [ 0.220313] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227442] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.248969] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.252150] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.256029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268000] acpiphp: Slot [2] registered [ 0.269412] acpiphp: Slot [5] registered [ 0.271392] acpiphp: Slot [6] registered [ 0.274227] acpiphp: Slot [7] registered [ 0.276322] acpiphp: Slot [8] registered [ 0.278271] acpiphp: Slot [9] registered [ 0.281240] acpiphp: Slot [10] registered [ 0.284074] acpiphp: Slot [3] registered [ 0.285206] acpiphp: Slot [4] registered [ 0.287268] acpiphp: Slot [11] registered [ 0.290206] acpiphp: Slot [12] registered [ 0.292145] acpiphp: Slot [13] registered [ 0.294162] acpiphp: Slot [14] registered [ 0.297254] acpiphp: Slot [15] registered [ 0.299126] acpiphp: Slot [16] registered [ 0.301222] acpiphp: Slot [17] registered [ 0.303194] acpiphp: Slot [18] registered [ 0.306174] acpiphp: Slot [19] registered [ 0.308170] acpiphp: Slot [20] registered [ 0.310744] acpiphp: Slot [21] registered [ 0.311148] acpiphp: Slot [22] registered [ 0.313171] acpiphp: Slot [23] registered [ 0.315168] acpiphp: Slot [24] registered [ 0.317143] acpiphp: Slot [25] registered [ 0.319138] acpiphp: Slot [26] registered [ 0.321158] acpiphp: Slot [27] registered [ 0.323398] acpiphp: Slot [28] registered [ 0.325144] acpiphp: Slot [29] registered [ 0.328126] acpiphp: Slot [30] registered [ 0.330153] acpiphp: Slot [31] registered [ 0.332178] PCI host bridge to bus 0000:00 [ 0.334032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340927] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.344039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349044] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355376] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.360550] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365739] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378042] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.383733] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.387028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.391031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.395029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.400378] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.403998] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.409052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.414738] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 10742 usecs [ 0.420193] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.428052] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.449022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.458092] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.467087] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.486028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.505000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.528021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.537274] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.551020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.561021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.586020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.602353] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.609026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.613020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.632020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.647913] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.655023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.659020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.675022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.706157] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.714025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.721000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.750029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.764000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.771021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.783021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.801031] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.819000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.819571] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.821547] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.824457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.827311] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.832152] iommu: Default domain type: Passthrough [ 0.834426] SCSI subsystem initialized [ 0.836202] ACPI: bus type USB registered [ 0.839000] usbcore: registered new interface driver usbfs [ 0.840150] usbcore: registered new interface driver hub [ 0.845115] usbcore: registered new device driver usb [ 0.849246] pps_core: LinuxPPS API ver. 1 registered [ 0.854017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.861090] PTP clock support registered [ 0.865099] EDAC MC: Ver: 3.0.0 [ 0.867220] PCI: Using ACPI for IRQ routing [ 0.870000] NetLabel: Initializing [ 0.873024] NetLabel: domain hash size = 128 [ 0.876029] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.879119] NetLabel: unlabeled traffic allowed by default [ 0.881065] vgaarb: loaded [ 0.882330] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.885027] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.905605] clocksource: Switched to clocksource kvm-clock [ 1.266114] VFS: Disk quotas dquot_6.6.0 [ 1.268376] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.271314] *** VALIDATE ramfs *** [ 1.272830] *** VALIDATE hugetlbfs *** [ 1.274831] pnp: PnP ACPI init [ 1.278263] pnp: PnP ACPI: found 6 devices [ 1.298306] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.304752] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.308426] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.311073] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.314354] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.317900] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.321404] NET: Registered protocol family 2 [ 1.325050] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.333702] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.339264] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.347085] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.352301] TCP: Hash tables configured (established 65536 bind 65536) [ 1.357193] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.362316] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.366716] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.371623] NET: Registered protocol family 1 [ 1.375618] RPC: Registered named UNIX socket transport module. [ 1.378433] RPC: Registered udp transport module. [ 1.381587] RPC: Registered tcp transport module. [ 1.383282] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.387641] NET: Registered protocol family 44 [ 1.390416] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.393276] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.396325] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.400230] PCI: CLS 0 bytes, default 64 [ 1.402646] Unpacking initramfs... [ 4.272732] debug: unmapping init [mem 0xffff99a87cc54000-0xffff99a87ffbffff] [ 4.278411] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.281601] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.289477] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 5.369616] Initialise system trusted keyrings [ 5.373541] Key type blacklist registered [ 5.376530] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.392755] zbud: loaded [ 5.396736] *** VALIDATE nfs *** [ 5.398592] *** VALIDATE nfs4 *** [ 5.400770] pstore: using deflate compression [ 5.406235] Platform Keyring initialized [ 5.567652] NET: Registered protocol family 38 [ 5.570338] Key type asymmetric registered [ 5.572108] Asymmetric key parser 'x509' registered [ 5.574616] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.580610] io scheduler mq-deadline registered [ 5.585716] io scheduler kyber registered [ 5.588488] io scheduler bfq registered [ 5.595553] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.602350] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.608205] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.615324] ACPI: Power Button [PWRF] [ 5.626565] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.646060] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.674164] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.707316] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.769429] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.833907] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.873129] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.886398] Non-volatile memory driver v1.3 [ 5.892344] Linux agpgart interface v0.103 [ 5.946652] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.954443] vda: detected capacity change from 0 to 74694656 [ 5.989200] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.992907] vdb: detected capacity change from 0 to 1073741824 [ 6.030702] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.038432] vdc: detected capacity change from 0 to 2621440000 [ 6.093277] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.107470] vdd: detected capacity change from 0 to 2621440000 [ 6.168511] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.177761] vde: detected capacity change from 0 to 4294967296 [ 6.235763] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.244460] vdf: detected capacity change from 0 to 4294967296 [ 6.268341] libphy: Fixed MDIO Bus: probed [ 6.340516] usbcore: registered new interface driver usbserial_generic [ 6.349947] usbserial: USB Serial support registered for generic [ 6.359411] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.380086] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.387940] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.392981] mousedev: PS/2 mouse device common for all mice [ 6.402297] rtc_cmos 00:05: RTC can wake from S4 [ 6.411228] rtc_cmos 00:05: registered as rtc0 [ 6.413513] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.417567] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.423045] intel_pstate: CPU model not supported [ 6.442739] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.451359] hid: raw HID events driver (C) Jiri Kosina [ 6.456779] usbcore: registered new interface driver usbhid [ 6.462192] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.465299] usbhid: USB HID core driver [ 6.465545] drop_monitor: Initializing network drop monitor service [ 6.465704] Initializing XFRM netlink socket [ 6.472380] NET: Registered protocol family 10 [ 6.493760] Segment Routing with IPv6 [ 6.497494] NET: Registered protocol family 17 [ 6.502631] mpls_gso: MPLS GSO support [ 6.512270] RAS: Correctable Errors collector initialized. [ 6.515757] AVX version of gcm_enc/dec engaged. [ 6.519520] AES CTR mode by8 optimization enabled [ 6.827760] sched_clock: Marking stable (6827458142, 0)->(8200106883, -1372648741) [ 6.835739] registered taskstats version 1 [ 6.838429] Loading compiled-in X.509 certificates [ 6.844221] zswap: loaded using pool lzo/zbud [ 6.890561] Key type big_key registered [ 6.917682] Key type encrypted registered [ 6.920711] ima: No TPM chip found, activating TPM-bypass! [ 6.924750] ima: Allocated hash algorithm: sha1 [ 6.928739] ima: No architecture policies found [ 6.933332] evm: Initialising EVM extended attributes: [ 6.936422] evm: security.selinux [ 6.938550] evm: security.ima [ 6.940706] evm: security.capability [ 6.942626] evm: HMAC attrs: 0x1 [ 6.949546] rtc_cmos 00:05: setting system clock to 2026-08-11 02:22:37 UTC (1786414957) [ 6.960385] debug: unmapping init [mem 0xffffffffbaa03000-0xffffffffbabfffff] [ 6.965501] debug: unmapping init [mem 0xffffffffb9782000-0xffffffffb9a58fff] [ 6.975211] Write protecting the kernel read-only data: 28672k [ 6.980585] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 6.985430] debug: unmapping init [mem 0xffffffffb8714000-0xffffffffb87fffff] [ 7.061500] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.077125] systemd[1]: Detected virtualization kvm. [ 7.079975] systemd[1]: Detected architecture x86-64. [ 7.081667] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.110370] systemd[1]: No hostname configured. [ 7.113129] systemd[1]: Set hostname to . [ 7.116953] random: systemd: uninitialized urandom read (16 bytes read) [ 7.120481] systemd[1]: Initializing machine ID from random generator. [ 7.504627] random: systemd: uninitialized urandom read (16 bytes read) [ 7.511661] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 7.521217] random: systemd: uninitialized urandom read (16 bytes read) [ 7.525138] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.539679] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.804664] device-mapper: uevent: version 1.0.3 [ 10.818050] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.265769] random: fast init done [ 13.320321] virtio_net virtio0 ens2: renamed from eth0 [ 13.821257] scsi host0: ata_piix [ 13.966921] scsi host1: ata_piix [ 13.980172] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.012627] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.466835] random: crng init done [ 19.468158] random: 7 urandom warning(s) missed due to ratelimiting [ 22.049263] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 24.639706] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 28.778701] printk: systemd: 26 output lines suppressed due to ratelimiting [ 29.473723] SELinux: Disabled at runtime. [ 29.612809] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.634132] systemd[1]: Detected virtualization kvm. [ 29.638970] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 31.834818] systemd[1]: initrd-switch-root.service: Succeeded. [ 31.843868] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 31.858829] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 31.872213] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.882754] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.895271] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.908880] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ 32.945087] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 35.078310] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 36.853362] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.260072] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.246611] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.534369] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [* ] A start job is running for Configur…only root support (16s / no limit)[ 48.682832] Key type dns_resolver registered [** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit)[ 49.689481] NFS: Registering the id_resolver key type [ 49.696561] Key type id_resolver registered [ 49.701408] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ 51.325193] hrtimer: interrupt took 2908192 ns [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg450-server login: [ 141.010903] libcfs: loading out-of-tree module taints kernel. [ 141.167398] Key type ._llcrypt registered [ 141.172840] Key type .llcrypt registered [ 141.384139] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 166.224983] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 169.580170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 169.611177] alg: No test for adler32 (adler32-zlib) [ 171.255376] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 172.693492] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 174.591295] Key type lgssc registered [ 178.150286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 202.530117] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 264.570458] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 288.149267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 288.252137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 289.767854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 289.824844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 289.971056] Lustre: lustre-MDT0000: new disk, initializing [ 290.145340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 290.182306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 296.533698] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 314.860543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 315.094407] Lustre: 6543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 315.150782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 315.158599] Lustre: Skipped 1 previous similar message [ 315.246562] Lustre: lustre-MDT0001: new disk, initializing [ 315.320211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 315.356567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 315.379068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 321.954916] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 327.612936] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 340.606607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 340.994679] Lustre: lustre-OST0000: new disk, initializing [ 341.005240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 341.010623] Lustre: 8485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 341.084871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.621319] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 351.306362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 351.334595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 351.434967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 368.529298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 368.749415] Lustre: lustre-OST0001: new disk, initializing [ 368.757411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 368.766648] Lustre: 9559:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 368.898141] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 377.414875] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 378.382385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 378.389273] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 378.452974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 393.763681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 402.281943] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 411.537130] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing check_logdir /tmp/testlogs/ [ 418.432198] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing yml_node [ 423.929589] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 428.214661] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 432.108788] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 434.686893] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 22:29:42 EDT 2026 [ 457.180771] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 459.878155] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 479.202319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 479.206305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.223164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 480.744344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 480.748523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.057742] Lustre: server umount lustre-MDT0000 complete [ 489.308966] LustreError: 6537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415439 with bad export cookie 11155645695046974756 [ 489.318089] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.333471] LustreError: 6537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 490.977698] LustreError: 12303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.981895] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.989100] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 490.989116] Lustre: Skipped 3 previous similar messages [ 491.017197] LustreError: 12303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 491.067220] Lustre: Skipped 3 previous similar messages [ 495.010494] LustreError: 8695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.018154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 495.049638] LustreError: 8695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 495.060371] Lustre: Skipped 2 previous similar messages [ 495.839380] Lustre: server umount lustre-MDT0001 complete [ 501.219450] Lustre: server umount lustre-OST0000 complete [ 506.343583] Lustre: server umount lustre-OST0001 complete [ 515.290727] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 526.643659] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 563.524947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.783514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 563.841384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 563.943689] Lustre: lustre-MDT0000: new disk, initializing [ 564.037464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.059303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 569.687921] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 580.876433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.098339] Lustre: 15117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 581.161753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 581.165334] Lustre: Skipped 1 previous similar message [ 581.230613] Lustre: lustre-MDT0001: new disk, initializing [ 581.363649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 581.445981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 581.471410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 587.339615] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 599.368895] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 607.324509] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 615.309136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 615.766797] Lustre: lustre-OST0000: new disk, initializing [ 615.776441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 615.793522] Lustre: 17086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 616.013420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.343471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 617.363645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 617.764301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 625.728937] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 635.712861] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 640.894185] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 641.280947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 646.492592] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 646.907634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 653.282496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 653.291994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.305370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.460807] Lustre: server umount lustre-OST0000 complete [ 663.524080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 663.536095] LustreError: Skipped 1 previous similar message [ 663.546426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.557456] Lustre: Skipped 1 previous similar message [ 663.562224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 663.567564] Lustre: Skipped 1 previous similar message [ 665.785441] Lustre: server umount lustre-MDT0000 complete [ 669.161536] LustreError: 15127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 669.187753] LustreError: 15127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 669.864697] LustreError: 15107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415620 with bad export cookie 11155645695046976289 [ 669.873163] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.878834] LustreError: 15107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 670.197723] Lustre: server umount lustre-MDT0001 complete [ 674.622409] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:33:43 (1786415623) [ 677.392516] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 679.443416] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:33:47 (1786415627) [ 686.366877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.041422] LustreError: 19498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.120058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 692.216623] LustreError: 19499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 692.414397] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 702.382993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.433238] LustreError: 19498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.463063] LustreError: 19498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 707.470320] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 715.947890] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 722.781672] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 730.165295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.340205] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 751.357449] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 762.339717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 762.352390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.367542] Lustre: Skipped 1 previous similar message [ 762.379373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 762.390091] Lustre: Skipped 2 previous similar messages [ 767.214971] Lustre: server umount lustre-OST0000 complete [ 786.399175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 786.610078] Lustre: server umount lustre-MDT0000 complete [ 788.973402] LustreError: 19495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 789.015963] LustreError: 19495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 790.907339] LustreError: 19480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415741 with bad export cookie 11155645695046977171 [ 790.910642] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 790.919561] LustreError: 19480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 791.352630] Lustre: server umount lustre-MDT0001 complete [ 804.399193] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 808.063574] Key type lgssc unregistered [ 808.411825] LNet: 23036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.419567] LNetError: 23036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.432988] LNet: Removed LNI 192.168.204.150@tcp [ 809.905417] Key type .llcrypt unregistered [ 809.909862] Key type ._llcrypt unregistered [ 842.408242] Key type ._llcrypt registered [ 842.410256] Key type .llcrypt registered [ 842.589933] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 843.695778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 843.788452] alg: No test for adler32 (adler32-zlib) [ 844.969785] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 845.368547] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 847.031372] Key type lgssc registered [ 848.162750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 856.963332] loop: module loaded [ 857.069173] loop0: detected capacity change from 0 to 204800000 [ 857.263096] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 857.272431] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 857.500470] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 863.201202] loop0: detected capacity change from 0 to 204800000 [ 863.422799] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 863.432812] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 863.445964] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 868.877254] loop0: detected capacity change from 0 to 204800000 [ 874.733306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 874.753199] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.042152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 876.146712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 876.224300] Lustre: lustre-MDT0000: new disk, initializing [ 876.310524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.328639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 881.172449] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 889.684187] loop1: detected capacity change from 0 to 204800000 [ 897.126611] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.553904] Lustre: lustre-OST07c6: new disk, initializing [ 897.557438] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 897.563567] Lustre: Skipped 1 previous similar message [ 897.575762] Lustre: 26223:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 897.695180] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 906.987265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 907.002452] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 907.080276] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 907.323363] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 913.960423] Lustre: 25210:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 913.973347] Lustre: 25210:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 913.977671] Lustre: 25210:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 913.985897] Lustre: 25210:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 913.992647] Lustre: 25210:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 913.998289] Lustre: 25210:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 927.478915] Lustre: 25209:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 927.487270] Lustre: 25209:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 927.494522] Lustre: 25209:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 927.502425] Lustre: 25209:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 927.512553] Lustre: 25209:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 927.525455] Lustre: 25209:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 932.284543] Lustre: 25210:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 932.296067] Lustre: 25210:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 932.302846] Lustre: 25210:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 932.313478] Lustre: 25210:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 932.325557] Lustre: 25210:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 932.336030] Lustre: 25210:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 932.346099] Lustre: 25210:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 932.356326] Lustre: 25210:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 932.366320] Lustre: 25210:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 932.380612] Lustre: 25210:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 932.396386] Lustre: 25210:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 932.408855] Lustre: 25210:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 935.648926] Lustre: 25211:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 935.662303] Lustre: 25211:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 935.668082] Lustre: 25211:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 935.681025] Lustre: 25211:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 935.693189] Lustre: 25211:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 935.699187] Lustre: 25211:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 935.704708] Lustre: 25211:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 935.710462] Lustre: 25211:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 935.715748] Lustre: 25211:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 935.723314] Lustre: 25211:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 935.726870] Lustre: 25211:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 935.731271] Lustre: 25211:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 940.220910] Lustre: 25210:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 940.238981] Lustre: 25210:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 940.247775] Lustre: 25210:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 940.253743] Lustre: 25210:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 940.260729] Lustre: 25210:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 940.268481] Lustre: 25210:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 940.280924] Lustre: 25210:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 940.291605] Lustre: 25210:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 940.294374] Lustre: 25210:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 940.301761] Lustre: 25210:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 940.310366] Lustre: 25210:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 940.319235] Lustre: 25210:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 951.780983] Lustre: 25210:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 951.789331] Lustre: 25210:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 951.794622] Lustre: 25210:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 951.798977] Lustre: 25210:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 951.802772] Lustre: 25210:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 951.806839] Lustre: 25210:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 951.810349] Lustre: 25210:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 951.818173] Lustre: 25210:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 951.826533] Lustre: 25210:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 951.834809] Lustre: 25210:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 951.842837] Lustre: 25210:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 951.852769] Lustre: 25210:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 999.396144] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.409180] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1004.248354] Lustre: server umount lustre-OST07c6 complete [ 1014.594972] Lustre: server umount lustre-MDT0000 complete [ 1031.471817] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1033.689482] Key type lgssc unregistered [ 1033.953673] LNet: 28786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.962211] LNetError: 28786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.983345] LNet: Removed LNI 192.168.204.150@tcp [ 1034.697377] Key type .llcrypt unregistered [ 1034.698925] Key type ._llcrypt unregistered [ 1053.025575] Key type ._llcrypt registered [ 1053.029873] Key type .llcrypt registered [ 1053.137544] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1054.485780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.543158] alg: No test for adler32 (adler32-zlib) [ 1055.578199] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1055.867936] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1057.551177] Key type lgssc registered [ 1059.056411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1067.188587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1067.211588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.550549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.735772] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1078.756692] LustreError: 30059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.788207] LustreError: 30059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1081.336577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.688089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1087.521302] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1095.599607] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.764357] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.108235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.637795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.729788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1117.887913] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1126.955764] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1142.762077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.771749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.781762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.804708] Lustre: Skipped 1 previous similar message [ 1142.834566] Lustre: Skipped 1 previous similar message [ 1147.890746] LustreError: 31882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.042601] Lustre: server umount lustre-OST0000 complete [ 1152.995110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.998846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.009443] Lustre: Skipped 1 previous similar message [ 1153.024205] Lustre: Skipped 1 previous similar message [ 1158.112777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.118222] Lustre: Skipped 1 previous similar message [ 1163.233273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.241777] Lustre: Skipped 1 previous similar message [ 1167.327155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1167.527220] Lustre: server umount lustre-MDT0000 complete [ 1168.358932] LustreError: 31635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1168.387771] LustreError: 31635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1172.084044] LustreError: 30041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416122 with bad export cookie 2988625361083349474 [ 1172.090433] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1172.098826] LustreError: 30041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1172.346593] Lustre: server umount lustre-MDT0001 complete [ 1185.574801] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1189.078696] Key type lgssc unregistered [ 1189.375676] LNet: 33800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.387340] LNetError: 33800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.439384] LNet: Removed LNI 192.168.204.150@tcp [ 1191.867818] Key type .llcrypt unregistered [ 1191.872664] Key type ._llcrypt unregistered [ 1215.449949] Key type ._llcrypt registered [ 1215.456881] Key type .llcrypt registered [ 1215.714092] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1217.223502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1217.412561] alg: No test for adler32 (adler32-zlib) [ 1218.919641] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1219.158955] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1220.887163] Key type lgssc registered [ 1222.542490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1232.110174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1232.144854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1233.575373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.048219] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1244.131644] LustreError: 35070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.177563] LustreError: 35070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1249.258729] LustreError: 35071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.350223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.027549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1257.790350] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1267.493309] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1274.574363] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1283.666796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.148957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.635452] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1302.405238] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1303.617716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1318.885642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1318.901196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.914558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1318.927642] Lustre: Skipped 1 previous similar message [ 1323.163446] Lustre: server umount lustre-OST0000 complete [ 1327.587902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1327.601918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.629059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1327.636216] Lustre: Skipped 1 previous similar message [ 1329.124443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.126199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1329.144631] Lustre: Skipped 1 previous similar message [ 1333.578382] Lustre: server umount lustre-MDT0000 complete [ 1334.242582] LustreError: 35801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.265046] LustreError: 35801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1338.463075] LustreError: 35051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416289 with bad export cookie 57040710388963817 [ 1338.470572] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1338.481155] LustreError: 35051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1339.034496] Lustre: server umount lustre-MDT0001 complete [ 1354.796486] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1358.875816] Key type lgssc unregistered [ 1359.293334] LNet: 38637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.305190] LNetError: 38637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.326565] LNet: Removed LNI 192.168.204.150@tcp [ 1360.648738] Key type .llcrypt unregistered [ 1360.651249] Key type ._llcrypt unregistered [ 1389.793152] Key type ._llcrypt registered [ 1389.801194] Key type .llcrypt registered [ 1389.928950] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1391.157951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1391.244249] alg: No test for adler32 (adler32-zlib) [ 1392.573696] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1392.790580] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1394.487288] Key type lgssc registered [ 1395.566864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1404.670894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1404.717730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.182723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.468395] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1415.678517] LustreError: 40055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.705773] LustreError: 40055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1419.833204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.123479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1424.809281] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1433.292258] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1439.885549] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1449.002086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1449.341236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.403702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1456.671966] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1465.446534] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1471.727339] Lustre: server umount lustre-MDT0000 complete [ 1472.662211] LustreError: 40049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.052293] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.059343] LustreError: 40049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.078752] Lustre: Skipped 2 previous similar messages [ 1475.118402] LustreError: 40049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1475.821325] LustreError: 40034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416426 with bad export cookie 2433606423500682809 [ 1475.845565] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1475.859020] LustreError: 40034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1477.757618] LustreError: 42012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.778588] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.50@tcp (stopping) [ 1477.778783] LustreError: 42012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1478.330910] Lustre: server umount lustre-MDT0001 complete [ 1510.417547] Lustre: server umount lustre-OST0000 complete [ 1530.349787] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1533.685637] Key type lgssc unregistered [ 1534.138628] LNet: 43920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.154402] LNetError: 43920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.208279] LNet: Removed LNI 192.168.204.150@tcp [ 1536.187263] Key type .llcrypt unregistered [ 1536.190359] Key type ._llcrypt unregistered [ 1569.638373] Key type ._llcrypt registered [ 1569.641818] Key type .llcrypt registered [ 1569.811687] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1571.945483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1572.033871] alg: No test for adler32 (adler32-zlib) [ 1573.655232] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1573.952530] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1575.625294] Key type lgssc registered [ 1577.147763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1589.026812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1589.081483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.696977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.074305] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1601.000263] LustreError: 45341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.027774] LustreError: 45341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1606.120678] LustreError: 45340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.813862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1607.055803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1612.369996] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1621.522317] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1627.602508] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1635.147590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1635.445698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.485218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1644.100961] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1653.906248] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1661.078826] Lustre: lustre-OST0000: Not available for connect from 192.168.204.50@tcp (stopping) [ 1662.435432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1662.442413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.460638] LustreError: Skipped 1 previous similar message [ 1662.461610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1662.501689] Lustre: Skipped 1 previous similar message [ 1666.172522] Lustre: lustre-OST0000: Not available for connect from 192.168.204.50@tcp (stopping) [ 1666.182670] Lustre: Skipped 1 previous similar message [ 1666.558608] Lustre: server umount lustre-OST0000 complete [ 1688.036448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.052663] Lustre: Skipped 1 previous similar message [ 1688.055065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.071778] Lustre: Skipped 1 previous similar message [ 1692.898928] Lustre: server umount lustre-MDT0000 complete [ 1693.163379] LustreError: 45340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.208229] LustreError: 46076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416647 with bad export cookie 4498234775002632211 [ 1697.222564] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1697.240326] LustreError: 46076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1703.668523] Lustre: server umount lustre-MDT0001 complete [ 1716.946988] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1719.968279] Key type lgssc unregistered [ 1720.495454] LNet: 49057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.507926] LNetError: 49057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.526758] LNet: Removed LNI 192.168.204.150@tcp [ 1721.561424] Key type .llcrypt unregistered [ 1721.568956] Key type ._llcrypt unregistered [ 1751.501830] Key type ._llcrypt registered [ 1751.503633] Key type .llcrypt registered [ 1751.723140] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1753.382827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1753.411235] alg: No test for adler32 (adler32-zlib) [ 1755.465302] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1755.712864] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1757.399602] Key type lgssc registered [ 1759.076586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1768.696359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1768.714607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.195336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.655662] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1779.689363] LustreError: 50472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1779.702721] LustreError: 50472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1784.801442] LustreError: 50473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.360655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.698403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1791.062601] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1801.102705] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1808.919768] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1816.718060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1816.988817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.336662] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1830.856722] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1840.575285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1848.728499] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1852.851182] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1865.448392] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1022h53m33s [ 1867.490974] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1871.338429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.342065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.364181] Lustre: Skipped 2 previous similar messages [ 1871.372899] Lustre: Skipped 1 previous similar message [ 1872.429681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.437831] Lustre: Skipped 3 previous similar messages [ 1873.561184] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.50@tcp (stopping) [ 1875.622928] Lustre: server umount lustre-MDT0000 complete [ 1878.501599] LustreError: 51451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1878.536793] LustreError: 51451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1878.875532] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1880.698958] LustreError: 50468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.890456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.996659] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1886.285901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.365212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1887.356424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1892.263730] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1896.056943] Lustre: DEBUG MARKER: Wait for df (36778) ... [ 1897.637484] Lustre: DEBUG MARKER: done [ 1901.537737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.546654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.556820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.560675] Lustre: Skipped 1 previous similar message [ 1903.079626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.082160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1907.668142] Lustre: server umount lustre-OST0000 complete [ 1913.314668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.318489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1913.343591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.366180] Lustre: Skipped 3 previous similar messages [ 1918.475771] Lustre: server umount lustre-MDT0000 complete [ 1920.485357] LustreError: 51200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1920.519694] LustreError: 51200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1921.975915] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416872 with bad export cookie 386769202294015654 [ 1921.976734] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.013439] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1922.368823] Lustre: server umount lustre-MDT0001 complete [ 1934.953728] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1938.275471] Key type lgssc unregistered [ 1938.650846] LNet: 55423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.664680] LNetError: 55423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.697687] LNet: Removed LNI 192.168.204.150@tcp [ 1940.284572] Key type .llcrypt unregistered [ 1940.288392] Key type ._llcrypt unregistered [ 1982.122786] Key type ._llcrypt registered [ 1982.124710] Key type .llcrypt registered [ 1982.236956] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1983.235768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1983.261208] alg: No test for adler32 (adler32-zlib) [ 1984.246725] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1984.444630] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1986.103256] Key type lgssc registered [ 1987.647778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1997.010133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1997.034451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.420364] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1998.451340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1998.640331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.746929] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2014.693383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.775378] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2014.811649] Lustre: 57635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2015.255986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.634748] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2029.543397] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2037.633704] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2045.484585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2045.669243] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2045.830347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.443225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2053.748868] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2062.397478] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2071.207905] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1022h56m59s [ 2073.237821] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2080.108935] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2082.388607] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2082.616923] LustreError: 57639:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2102.948718] Lustre: lustre-MDT0000: Client a52d355c-0f26-42fc-8093-d6d3fb34acd9 (at 192.168.204.50@tcp) reconnecting [ 2122.687115] LustreError: 57639:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2125.998844] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2129.048604] Lustre: DEBUG MARKER: done [ 2138.592412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2138.598325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.599646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2138.629687] Lustre: Skipped 1 previous similar message [ 2143.728762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2143.738034] Lustre: Skipped 2 previous similar messages [ 2148.839047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2148.844943] Lustre: Skipped 1 previous similar message [ 2150.879250] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2151.071835] Lustre: server umount lustre-OST0000 complete [ 2158.561887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2158.568313] LustreError: Skipped 1 previous similar message [ 2158.572456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.587140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2161.869835] Lustre: server umount lustre-MDT0000 complete [ 2165.325184] LustreError: 57625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417115 with bad export cookie 13826471322722098283 [ 2165.347547] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.356141] LustreError: 57625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2165.731566] Lustre: server umount lustre-MDT0001 complete [ 2178.088902] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2181.049940] Key type lgssc unregistered [ 2181.467041] LNet: 61968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.474000] LNetError: 61968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.492185] LNet: Removed LNI 192.168.204.150@tcp [ 2182.770672] Key type .llcrypt unregistered [ 2182.773431] Key type ._llcrypt unregistered [ 2236.438658] Key type ._llcrypt registered [ 2236.445888] Key type .llcrypt registered [ 2236.679567] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2237.976419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.464248] alg: No test for adler32 (adler32-zlib) [ 2239.740977] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2240.191949] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2241.959308] Key type lgssc registered [ 2243.551239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2247.525573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.540157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.831891] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2248.854363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2248.992776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.213132] Lustre: Failing over lustre-MDT0000 [ 2250.438845] Lustre: server umount lustre-MDT0000 complete [ 2258.354489] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:00:06 (1786417206) [ 2265.359327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.724326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.443930] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2280.374325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.507913] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2280.545218] Lustre: 64768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2280.749996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.885722] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2293.474952] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2299.379993] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2306.772972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.925802] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2307.095385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.986847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2314.313211] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2321.737371] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2327.798042] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2333.672395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.687583] Lustre: Skipped 2 previous similar messages [ 2333.775675] Lustre: server umount lustre-MDT0000 complete [ 2337.585680] LustreError: 64763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417288 with bad export cookie 13654717847058533191 [ 2337.591933] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.598744] LustreError: 64763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2337.845845] Lustre: server umount lustre-MDT0001 complete [ 2341.280833] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2345.786645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2354.538789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.911504] Lustre: 63694:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417289/real 1786417289] req@00000000269970e5 x1873194191459968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417305 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2354.939063] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.199164] Lustre: 63695:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417293/real 1786417293] req@000000004bb547dd x1873194191460096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2363.169390] LustreError: 63692:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004a5179ed x1873194191461120/t0(0) o250->MGC192.168.204.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2363.492027] LustreError: 66617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.504252] LustreError: 66617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2363.610983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.710854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2364.576231] Lustre: 63693:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417298/real 1786417298] req@000000002dfd6ae2 x1873194191460480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2364.605643] Lustre: 63693:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2364.643598] LustreError: 68553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.656874] LustreError: 68553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2368.385913] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2368.736586] Lustre: 63693:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417303/real 1786417303] req@000000002ab12543 x1873194191460608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417319 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2370.024724] LustreError: 68552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.047493] LustreError: 68552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2374.118243] LustreError: 68645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.132548] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2374.158524] LustreError: 68645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2377.092562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.466905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2382.233563] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2382.828821] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2391.558599] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2398.549794] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2408.425689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.430751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2408.443253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2408.448448] Lustre: Skipped 2 previous similar messages [ 2408.456423] Lustre: Skipped 1 previous similar message [ 2411.644643] Lustre: server umount lustre-MDT0000 complete [ 2413.541020] LustreError: 68551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.561992] LustreError: 68551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2415.686138] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417366 with bad export cookie 13654717847058535466 [ 2415.693313] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2415.700614] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2416.018757] Lustre: server umount lustre-MDT0001 complete [ 2422.876521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2433.619434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.039263] Lustre: 63694:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417369/real 1786417369] req@00000000aee5d0b5 x1873194191488000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417385 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2435.068267] Lustre: 63694:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2435.082718] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.254504] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbd7f4d893e1c598d [ 2441.274138] Lustre: MGC192.168.204.150@tcp: Connection restored to 0@lo (at 0@lo) [ 2441.552085] LustreError: 66618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.625699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.685269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2446.423944] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2446.690056] Lustre: 63694:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417381/real 1786417381] req@00000000ea8bc6ad x1873194191488640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2446.723779] Lustre: 63694:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2456.366567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.051938] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2462.203368] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2471.163264] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2476.761176] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2483.108883] Lustre: server umount lustre-MDT0000 complete [ 2484.708372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.723066] LustreError: 71237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.740479] Lustre: Skipped 1 previous similar message [ 2484.766993] LustreError: 71237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2487.467694] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417438 with bad export cookie 13654717847058536845 [ 2487.476305] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.488258] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2487.864505] Lustre: server umount lustre-MDT0001 complete [ 2495.147347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2498.839807] Lustre: DEBUG MARKER: files compared the same [ 2511.705039] Lustre: server umount lustre-OST0000 complete [ 2529.680039] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2533.429489] Key type lgssc unregistered [ 2533.845661] LNet: 74513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2533.857580] LNetError: 74513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2533.874086] LNet: Removed LNI 192.168.204.150@tcp [ 2535.074817] Key type .llcrypt unregistered [ 2535.078232] Key type ._llcrypt unregistered [ 2564.498036] Key type ._llcrypt registered [ 2564.503658] Key type .llcrypt registered [ 2564.644666] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2566.795245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2567.036035] alg: No test for adler32 (adler32-zlib) [ 2568.175862] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2568.397379] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2570.071133] Key type lgssc registered [ 2572.197034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2580.398866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2580.411622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.981972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2587.571865] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug +malloc all [ 2592.227642] LustreError: 75930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.240261] LustreError: 75930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2597.203623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.344446] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2597.422696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2602.711761] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug +malloc all [ 2610.233213] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2616.350697] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2624.032799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.316817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.403578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2630.470423] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug +malloc all [ 2638.620875] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2652.129992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.136187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2652.142464] Lustre: Skipped 1 previous similar message [ 2652.159243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2657.249551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2657.254640] Lustre: Skipped 1 previous similar message [ 2657.515605] Lustre: server umount lustre-OST0000 complete [ 2662.371485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.373388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.385952] Lustre: Skipped 1 previous similar message [ 2667.491592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2667.500414] Lustre: Skipped 1 previous similar message [ 2667.832489] Lustre: server umount lustre-MDT0000 complete [ 2672.611341] LustreError: 75924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.651800] LustreError: 75924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2672.742100] LustreError: 75911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417623 with bad export cookie 80209905737206143 [ 2672.750600] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2672.769694] LustreError: 75911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2673.249834] Lustre: server umount lustre-MDT0001 complete [ 2688.517692] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2692.404676] Key type lgssc unregistered [ 2692.896774] LNet: 79489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2692.904946] LNetError: 79489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2692.926431] LNet: Removed LNI 192.168.204.150@tcp [ 2694.107221] Key type .llcrypt unregistered [ 2694.115402] Key type ._llcrypt unregistered [ 2728.879617] Key type ._llcrypt registered [ 2728.888629] Key type .llcrypt registered [ 2729.085946] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2730.857519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2730.968036] alg: No test for adler32 (adler32-zlib) [ 2732.323153] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2732.841395] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2734.615202] Key type lgssc registered [ 2736.254427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2746.245337] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2746.260108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.823819] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2766.943308] LustreError: 80889:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.150@tcp: failed processing log, type 4: rc = -110 [ 2793.631248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.233660] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2810.243419] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2820.967684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.056839] Lustre: *** cfs_fail_loc=706, val=0*** [ 2821.385789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.478510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2826.393353] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2826.738076] LustreError: 80914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.778888] LustreError: 80914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2827.756634] LustreError: 80915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.779756] LustreError: 80915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2832.866106] LustreError: 81975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.893375] LustreError: 81975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2835.194178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.757580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2841.559688] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2850.414580] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2857.057852] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2861.549797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.555625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2861.561418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.578501] Lustre: Skipped 1 previous similar message [ 2865.595789] Lustre: server umount lustre-OST0000 complete [ 2871.777994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.781767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.785040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2871.801220] Lustre: Skipped 1 previous similar message [ 2871.830208] Lustre: Skipped 2 previous similar messages [ 2875.731149] Lustre: server umount lustre-MDT0000 complete [ 2876.904133] LustreError: 81979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.935645] LustreError: 81979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2879.678753] LustreError: 80896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417830 with bad export cookie 15595019135653695732 [ 2879.696620] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2880.135249] Lustre: server umount lustre-MDT0001 complete [ 2893.426091] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2896.601683] Key type lgssc unregistered [ 2896.961857] LNet: 84518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2896.970811] LNetError: 84518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2896.985530] LNet: Removed LNI 192.168.204.150@tcp [ 2898.219759] Key type .llcrypt unregistered [ 2898.221557] Key type ._llcrypt unregistered [ 2931.046479] Key type ._llcrypt registered [ 2931.049048] Key type .llcrypt registered [ 2931.334790] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2932.830651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2933.025044] alg: No test for adler32 (adler32-zlib) [ 2934.356994] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2934.719195] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2936.535192] Key type lgssc registered [ 2937.630755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2947.160256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2947.174379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.786847] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2962.265101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.822195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2968.033218] LustreError: 86625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.051573] LustreError: 86625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2968.592695] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2973.158230] LustreError: 86623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.206290] LustreError: 86623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2978.198348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2978.291794] LustreError: 86629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.341110] LustreError: 86629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2978.787593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.908969] LustreError: 86628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.947055] LustreError: 86628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2986.091577] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2988.016716] LustreError: 86624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.048023] LustreError: 86624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2996.352982] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2996.381994] Lustre: MGS: Client 1155d607-6a8c-4de6-b7b5-55c95f794d33 (at 0@lo) reconnecting [ 2996.437179] LustreError: 86639:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2996.719927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2996.914234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3002.699112] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3018.722100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.732906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.751270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.311969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3022.320868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.335917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.364878] Lustre: Skipped 1 previous similar message [ 3024.654508] Lustre: server umount lustre-OST0000 complete [ 3032.550651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3032.556331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.560832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3032.589066] Lustre: Skipped 1 previous similar message [ 3034.568421] Lustre: server umount lustre-MDT0000 complete [ 3037.672938] LustreError: 86628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.711198] LustreError: 86628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3039.242337] Lustre: server umount lustre-MDT0001 complete [ 3043.643858] Lustre: server umount MGS complete [ 3060.506706] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:13:28 (1786418008) [ 3099.847512] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 3114.296517] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3155.365530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.242626] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3173.541572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.825437] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3173.894875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3173.973456] Lustre: lustre-MDT0001: new disk, initializing [ 3174.048516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3174.075691] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3174.091203] Lustre: 93822:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3174.111932] LustreError: 93822:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3179.256628] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3191.087033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.313879] Lustre: lustre-OST0000: new disk, initializing [ 3191.318377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3191.330539] Lustre: 94587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3191.426302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.514256] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3201.634107] Lustre: 95025:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000023352c97 x1873194920224512/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786418168 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3209.264684] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3217.908293] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3217.922898] Lustre: MGS: Client b59c7140-b5f6-417a-a733-84dfc0cf3f34 (at 0@lo) reconnecting [ 3217.997901] Lustre: 93090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3218.008602] LustreError: 93808:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3218.037523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3218.102377] Lustre: lustre-MDT0000: new disk, initializing [ 3218.196219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.223723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3218.238654] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3219.300937] Lustre: 95657:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000cc463e6 x1873194920231552/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786418185 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3223.625331] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3224.486457] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3228.458553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3228.481888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3228.559952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3250.663365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.665747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3250.695265] Lustre: Skipped 1 previous similar message [ 3250.708969] Lustre: Skipped 2 previous similar messages [ 3255.784707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3256.140302] Lustre: server umount lustre-OST0000 complete [ 3260.896983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.017838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.031587] Lustre: Skipped 4 previous similar messages [ 3266.784908] Lustre: server umount lustre-MDT0000 complete [ 3271.142269] LustreError: 93798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.171470] LustreError: 93798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3277.610857] Lustre: server umount lustre-MDT0001 complete [ 3282.390897] Lustre: server umount MGS complete [ 3295.550680] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:17:23 (1786418243) [ 3316.466478] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 3319.982561] Key type lgssc unregistered [ 3320.338307] LNet: 98160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3320.347215] LNetError: 98160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3320.371570] LNet: Removed LNI 192.168.204.150@tcp [ 3321.566251] Key type .llcrypt unregistered [ 3321.572231] Key type ._llcrypt unregistered [ 3342.718946] Key type ._llcrypt registered [ 3342.722736] Key type .llcrypt registered [ 3342.870471] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3343.989147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3344.013734] alg: No test for adler32 (adler32-zlib) [ 3345.443561] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 3345.744459] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 3347.480078] Key type lgssc registered [ 3348.777671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3358.225447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3358.243942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.869444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.503517] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3369.958375] LustreError: 99383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3370.013942] LustreError: 99383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3374.682487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.981159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.683604] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3388.668212] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3394.080889] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3400.430515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3400.861227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.985565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3408.183645] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3416.940465] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3430.890827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3430.894829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3430.906156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3430.914880] Lustre: Skipped 1 previous similar message [ 3434.561524] Lustre: server umount lustre-OST0000 complete [ 3441.127873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.135451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3441.147121] Lustre: Skipped 1 previous similar message [ 3441.152930] Lustre: Skipped 1 previous similar message [ 3445.513292] Lustre: server umount lustre-MDT0000 complete [ 3446.241713] LustreError: 99377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.263506] LustreError: 99377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3450.740075] LustreError: 99364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418401 with bad export cookie 17124789821267375358 [ 3450.751853] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3451.033761] Lustre: server umount lustre-MDT0001 complete [ 3464.258755] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 3468.487356] Key type lgssc unregistered [ 3468.839149] LNet: 102920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3468.850228] LNetError: 102920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3468.868964] LNet: Removed LNI 192.168.204.150@tcp [ 3469.849545] Key type .llcrypt unregistered [ 3469.852426] Key type ._llcrypt unregistered [ 3488.476862] Key type ._llcrypt registered [ 3488.479541] Key type .llcrypt registered [ 3488.594895] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3490.094401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3490.174931] alg: No test for adler32 (adler32-zlib) [ 3491.447336] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 3491.898993] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 3493.607179] Key type lgssc registered [ 3494.838814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3501.273901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3501.287353] LustreError: 103997:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3501.333969] LustreError: 104004:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3501.339271] LustreError: 103997:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4958 [ 3501.345766] LustreError: 103997:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3501.360970] LustreError: 103997:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3501.371681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.851498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.793321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.069182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3516.961389] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3522.794585] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing lsmod [ 3526.228377] LustreError: 105211:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3526.230183] LustreError: 105210:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3526.245276] LustreError: 105211:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 3526.270352] LustreError: 105211:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3526.292276] LustreError: 105211:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3526.303703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3527.048143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.109619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3531.744432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3531.751313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.768241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.281449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.283753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.296658] Lustre: Skipped 1 previous similar message [ 3533.310044] Lustre: Skipped 2 previous similar messages [ 3536.360480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.374456] Lustre: Skipped 2 previous similar messages [ 3536.908841] Lustre: server umount lustre-MDT0000 complete [ 3540.583520] LustreError: 104018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418491 with bad export cookie 10697802115698792651 [ 3540.585650] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3540.593213] LustreError: 104018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3541.472650] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.474729] LustreError: 104036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.489344] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3541.500288] LustreError: 104036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3542.559402] LustreError: 104472:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000072bf0e1a x1873195504261888/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3542.764866] Lustre: server umount lustre-MDT0001 complete [ 3546.859282] Lustre: server umount lustre-OST0000 complete [ 3556.154748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.589694] LustreError: 106272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.666776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.024479] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3561.959796] LustreError: 106273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.078564] LustreError: 106272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.658714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.021137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3575.520992] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3584.560103] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3591.861715] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3599.723557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3600.095299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.184811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 3607.122240] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3615.576148] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3622.882877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3622.891123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3622.910754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3625.960566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3628.058205] Lustre: server umount lustre-OST0000 complete [ 3636.192784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.206586] Lustre: Skipped 1 previous similar message [ 3636.212781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3636.219742] Lustre: Skipped 3 previous similar messages [ 3638.406705] Lustre: server umount lustre-MDT0000 complete [ 3641.313666] LustreError: 107228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.336187] LustreError: 107228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3643.091350] LustreError: 106254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418593 with bad export cookie 10697802115698793582 [ 3643.104712] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3643.110916] LustreError: 106254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3643.422900] Lustre: server umount lustre-MDT0001 complete [ 3656.738514] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 3659.921024] Key type lgssc unregistered [ 3660.247452] LNet: 109811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3660.257217] LNetError: 109811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3660.280646] LNet: Removed LNI 192.168.204.150@tcp [ 3661.669643] Key type .llcrypt unregistered [ 3661.671792] Key type ._llcrypt unregistered [ 3689.104510] Key type ._llcrypt registered [ 3689.107656] Key type .llcrypt registered [ 3689.364259] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3690.925388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3690.962418] alg: No test for adler32 (adler32-zlib) [ 3692.016699] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 3692.321433] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 3694.064218] Key type lgssc registered [ 3695.776689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3704.955487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3704.972523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.853527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3712.373386] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3717.095326] LustreError: 111225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3717.125907] LustreError: 111225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3722.225489] LustreError: 111224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3722.868309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.371446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3729.177966] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3738.080770] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3746.081945] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3754.499715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3755.053192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.553612] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3771.359737] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3777.610683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3786.798420] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3790.570864] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3798.005229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3798.013769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3798.035044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.588956] Lustre: server umount lustre-OST0000 complete [ 3808.230877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.236697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3808.251320] Lustre: Skipped 2 previous similar messages [ 3808.269772] Lustre: Skipped 2 previous similar messages [ 3812.926312] Lustre: server umount lustre-MDT0000 complete [ 3813.352279] LustreError: 111955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3813.376845] LustreError: 111955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3817.427525] LustreError: 113783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418767 with bad export cookie 2247122578310516969 [ 3817.434893] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3817.441681] LustreError: 113783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3817.921452] Lustre: server umount lustre-MDT0001 complete [ 3832.627480] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 3836.118272] Key type lgssc unregistered [ 3836.504270] LNet: 114896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3836.518224] LNetError: 114896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3836.545705] LNet: Removed LNI 192.168.204.150@tcp [ 3837.842654] Key type .llcrypt unregistered [ 3837.844565] Key type ._llcrypt unregistered [ 3857.311898] Key type ._llcrypt registered [ 3857.314703] Key type .llcrypt registered [ 3857.503960] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3859.141578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3859.221714] alg: No test for adler32 (adler32-zlib) [ 3860.328956] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 3860.604322] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 3862.265286] Key type lgssc registered [ 3864.800487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3876.321625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3876.338915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.937389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.883578] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3888.111225] LustreError: 116120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3888.159613] LustreError: 116120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3893.228184] LustreError: 116119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3897.335271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.651545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3903.804612] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3914.934714] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3922.338944] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3930.900177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.255647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3941.216781] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3947.686634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3952.424268] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3963.375123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3963.378097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.447632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3963.465160] Lustre: Skipped 1 previous similar message [ 3968.496877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3968.509357] Lustre: Skipped 1 previous similar message [ 3968.723350] Lustre: server umount lustre-OST0000 complete [ 3973.606600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.635803] Lustre: Skipped 1 previous similar message [ 3973.648878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3973.660639] Lustre: Skipped 1 previous similar message [ 3978.721880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3978.729687] Lustre: Skipped 1 previous similar message [ 3979.408226] Lustre: server umount lustre-MDT0000 complete [ 3983.841027] LustreError: 116119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3983.902376] LustreError: 116119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3984.449786] LustreError: 116099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418934 with bad export cookie 2759074419842246385 [ 3984.459702] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3984.770698] Lustre: server umount lustre-MDT0001 complete [ 4001.132859] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 4005.358399] Key type lgssc unregistered [ 4006.031874] LNet: 119660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4006.054631] LNetError: 119660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4006.091152] LNet: Removed LNI 192.168.204.150@tcp [ 4007.613634] Key type .llcrypt unregistered [ 4007.616165] Key type ._llcrypt unregistered [ 4036.269663] Key type ._llcrypt registered [ 4036.273710] Key type .llcrypt registered [ 4036.483631] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 4038.013335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4038.202388] alg: No test for adler32 (adler32-zlib) [ 4040.050165] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 4040.430827] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 4042.135186] Key type lgssc registered [ 4043.732995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4053.191378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4053.210526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.636630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.736834] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4065.266581] LustreError: 121082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4065.311195] LustreError: 121082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4070.369416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.380961] LustreError: 121081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.933894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4077.246724] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4086.045054] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4094.067879] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4102.350898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4102.902412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.491540] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4121.715893] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4125.498722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 4133.672455] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4138.900698] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4138.908955] Lustre: Skipped 2 previous similar messages [ 4142.822228] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4142.885707] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4143.940547] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4143.951808] Lustre: Skipped 2 previous similar messages [ 4151.511951] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4151.530053] Lustre: Skipped 1 previous similar message [ 4157.228675] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.50@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4157.246682] Lustre: Skipped 2 previous similar messages [ 4159.299365] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.50@tcp 192.168.0.[2,10]@tcp [ 4165.161516] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.50/24@tcp in log lustre-MDT0000 [ 4165.170257] Lustre: Skipped 2 previous similar messages [ 4174.393853] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.50/24@tcp [ 4174.406409] Lustre: Skipped 1 previous similar message [ 4181.992294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.004096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4182.016363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4182.028109] Lustre: Skipped 1 previous similar message [ 4183.782981] Lustre: server umount lustre-OST0000 complete [ 4188.591797] Lustre: server umount lustre-MDT0000 complete [ 4188.648381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4188.660817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.682284] LustreError: 121081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.227631] LustreError: 121082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.231798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.986800] LustreError: 121061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419143 with bad export cookie 6611303692643718369 [ 4192.989321] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4192.999759] LustreError: 121061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4195.475886] Lustre: server umount lustre-MDT0001 complete [ 4209.602471] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 4212.746120] Key type lgssc unregistered [ 4213.056410] LNet: 126080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4213.072310] LNetError: 126080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4213.092830] LNet: Removed LNI 192.168.204.150@tcp [ 4213.837431] Key type .llcrypt unregistered [ 4213.839450] Key type ._llcrypt unregistered [ 4247.606352] Key type ._llcrypt registered [ 4247.608712] Key type .llcrypt registered [ 4247.717618] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 4249.380671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4249.489301] alg: No test for adler32 (adler32-zlib) [ 4251.102021] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 4251.622905] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 4253.463215] Key type lgssc registered [ 4255.288936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4264.905713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4264.924854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4266.427617] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4266.433120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4266.478425] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.50/24@tcp [ 4272.408322] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4275.685638] LustreError: 127642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.718760] LustreError: 127642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4280.814122] LustreError: 127643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.723488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.166522] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4283.193487] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.50/24@tcp [ 4283.230294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4288.740790] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4297.204503] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4304.191786] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4312.136748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.705139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.345346] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4327.646283] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4336.276782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 4351.977338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4351.979789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.990685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4352.031188] Lustre: Skipped 1 previous similar message [ 4355.295558] Lustre: server umount lustre-OST0000 complete [ 4360.164762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4360.173552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.186646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4360.193105] Lustre: Skipped 1 previous similar message [ 4362.209758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4362.218484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.798771] Lustre: server umount lustre-MDT0000 complete [ 4367.331970] LustreError: 127637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.351041] LustreError: 127637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4369.989828] LustreError: 127624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419320 with bad export cookie 14059138269378808728 [ 4369.992565] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4370.000631] LustreError: 127624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4370.335887] Lustre: server umount lustre-MDT0001 complete [ 4383.756630] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 4387.146365] Key type lgssc unregistered [ 4387.508363] LNet: 131260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4387.514127] LNetError: 131260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4388.582777] LNet: Removed LNI 192.168.204.150@tcp [ 4390.093214] Key type .llcrypt unregistered [ 4390.099815] Key type ._llcrypt unregistered [ 4422.479819] Key type ._llcrypt registered [ 4422.482484] Key type .llcrypt registered [ 4422.596866] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 4441.110405] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 4443.505151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4443.590236] alg: No test for adler32 (adler32-zlib) [ 4444.617885] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 4445.018717] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 4446.743717] Key type lgssc registered [ 4448.245493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4486.699680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4486.742363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.902153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4487.925396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4487.983782] Lustre: lustre-MDT0000: new disk, initializing [ 4488.049610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.064788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4492.007490] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4503.513991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.743861] Lustre: 134661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4503.801298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4503.804349] Lustre: Skipped 1 previous similar message [ 4503.861739] Lustre: lustre-MDT0001: new disk, initializing [ 4503.942229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4503.973976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4503.978686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4508.430645] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4517.512732] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4523.636657] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4530.845776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4531.294991] Lustre: lustre-OST0000: new disk, initializing [ 4531.299219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4531.306791] Lustre: 136630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4531.393604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4536.931815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4536.951317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4537.174510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4538.045032] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4547.149680] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4551.786710] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4552.015917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4557.108151] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4557.349507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4562.920431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4562.926111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.928234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4562.934480] LustreError: Skipped 1 previous similar message [ 4562.972119] Lustre: Skipped 1 previous similar message [ 4565.319199] Lustre: server umount lustre-OST0000 complete [ 4570.596433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4570.610993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.633197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4570.637736] Lustre: Skipped 1 previous similar message [ 4573.157519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4573.162706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4576.183596] Lustre: server umount lustre-MDT0000 complete [ 4581.025465] LustreError: 134654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419531 with bad export cookie 4566296490958702425 [ 4581.041537] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.553563] Lustre: server umount lustre-MDT0001 complete [ 4589.956285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4590.267186] LustreError: 138847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.282470] LustreError: 138847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4590.352377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.276423] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4595.700329] LustreError: 138846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.779416] LustreError: 138847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.902713] LustreError: 138846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.993978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4605.526031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4611.189237] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4618.609947] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4624.066499] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4631.935911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4632.452717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.176027] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4649.180958] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4655.327495] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4655.691926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4661.657758] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4662.001213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4671.326293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4671.645590] Lustre: lustre-OST0001: new disk, initializing [ 4671.652126] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4671.660679] Lustre: 142219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4671.801782] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4679.767933] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4679.942718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4679.952381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4680.192517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4689.426304] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4689.860208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4695.677472] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4695.978854] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4702.791642] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d3087800800.ost_server_uuid 50 [ 4705.019904] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d3087800800.ost_server_uuid in IDLE state after 0 sec [ 4716.000444] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4716.013062] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.027620] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4716.029916] Lustre: Skipped 1 previous similar message [ 4716.049755] Lustre: Skipped 2 previous similar messages [ 4720.354200] Lustre: server umount lustre-OST0001 complete [ 4721.134413] LustreError: 140666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4721.162862] LustreError: 140666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4726.262478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4726.281941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4726.298721] Lustre: Skipped 1 previous similar message [ 4730.975451] Lustre: server umount lustre-OST0000 complete [ 4736.481744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.490773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.524131] Lustre: Skipped 2 previous similar messages [ 4741.874466] Lustre: server umount lustre-MDT0000 complete [ 4745.699444] LustreError: 138841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.710859] LustreError: 138841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4745.963562] LustreError: 138826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419696 with bad export cookie 4566296490958703307 [ 4745.965197] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4745.976455] LustreError: 138826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4746.503559] Lustre: server umount lustre-MDT0001 complete [ 4767.752477] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 4771.160817] Key type lgssc unregistered [ 4771.629907] LNet: 145228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4771.646198] LNetError: 145228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4771.679971] LNet: Removed LNI 192.168.204.150@tcp [ 4772.872308] Key type .llcrypt unregistered [ 4772.874497] Key type ._llcrypt unregistered [ 4815.749859] Key type ._llcrypt registered [ 4815.756092] Key type .llcrypt registered [ 4815.911663] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 4833.920954] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 4835.142021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4835.269510] alg: No test for adler32 (adler32-zlib) [ 4836.358756] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 4836.660403] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 4838.303416] Key type lgssc registered [ 4839.431864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4876.307792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4876.353715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4877.713759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4877.760816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4877.867439] Lustre: lustre-MDT0000: new disk, initializing [ 4878.021659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4878.061500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4883.661235] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4895.864031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4896.028460] Lustre: 149386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4896.110823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4896.114702] Lustre: Skipped 1 previous similar message [ 4896.174813] Lustre: lustre-MDT0001: new disk, initializing [ 4896.253255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4896.290538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4896.305965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4901.240850] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4911.344404] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4917.823543] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4926.144645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4926.425922] Lustre: lustre-OST0000: new disk, initializing [ 4926.433618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4926.453388] Lustre: 151354:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4926.572880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.418639] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4933.804320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4933.817587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4933.941393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4942.628959] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4947.002925] Lustre: Failing over lustre-OST0000 [ 4947.203263] Lustre: server umount lustre-OST0000 complete [ 4964.834586] Lustre: 147608:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786419899/real 1786419899] req@0000000093c165b9 x1873196914607488/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786419915 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4964.855645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.725424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4965.851960] Lustre: 147610:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786419900/real 1786419900] req@00000000398c82ee x1873196914608640/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786419916 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4965.880837] Lustre: 147610:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4965.886957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.906062] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4965.997595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.014041] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4966.049435] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4966.943143] Lustre: 147611:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786419900/real 1786419900] req@00000000a819d85f x1873196914607616/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786419916 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4966.977953] Lustre: 147611:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4971.052866] Lustre: 147610:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786419905/real 1786419905] req@00000000a9db32d0 x1873196914609664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786419921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4971.103600] Lustre: 147610:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4972.237300] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4975.242913] Lustre: lustre-OST0000: Recovery over after 0:09, of 3 clients 3 recovered and 0 were evicted. [ 4975.244944] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4975.878089] Lustre: Failing over lustre-MDT0000 [ 4976.204439] Lustre: server umount lustre-MDT0000 complete [ 4976.928270] Lustre: 147610:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786419910/real 1786419910] req@000000003aaeee06 x1873196914610816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786419926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4978.146492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4978.164561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.713648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.733874] Lustre: Skipped 1 previous similar message [ 4981.375159] LustreError: 149394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.387400] LustreError: 149394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4985.825089] LustreError: 149397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4985.840696] LustreError: 149397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4990.957185] LustreError: 149398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4990.972635] LustreError: 149398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4995.830940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.027972] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4996.066364] LustreError: 149397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.106408] LustreError: 149397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4996.461054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.544976] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4997.756472] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5000.652731] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5001.738153] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5001.748150] Lustre: Skipped 1 previous similar message [ 5001.803398] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5001.869644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5011.425944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5011.435619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5011.451538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5015.226501] Lustre: server umount lustre-OST0000 complete [ 5022.179531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.182472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5022.197676] Lustre: Skipped 1 previous similar message [ 5022.231028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.252968] Lustre: Skipped 2 previous similar messages [ 5025.950528] Lustre: server umount lustre-MDT0000 complete [ 5027.302875] LustreError: 149392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5027.334859] LustreError: 149392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5030.403127] LustreError: 149377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419980 with bad export cookie 12606291166282278090 [ 5030.414761] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5030.425317] LustreError: 149377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5030.920915] Lustre: server umount lustre-MDT0001 complete [ 5045.490580] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 5050.598454] Key type lgssc unregistered [ 5051.176171] LNet: 155010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5051.189254] LNetError: 155010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5051.217065] LNet: Removed LNI 192.168.204.150@tcp [ 5052.558286] Key type .llcrypt unregistered [ 5052.562636] Key type ._llcrypt unregistered [ 5097.293724] Key type ._llcrypt registered [ 5097.295503] Key type .llcrypt registered [ 5097.536039] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 5099.000911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5099.079295] alg: No test for adler32 (adler32-zlib) [ 5100.566495] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 5100.797367] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 5102.471220] Key type lgssc registered [ 5103.845456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5111.388312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5111.403691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.766174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5117.169894] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5123.054481] LustreError: 156535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.096896] LustreError: 156535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5127.199814] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5128.166820] LustreError: 156536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.007544] LustreError: 156564:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5132.021502] LustreError: 156564:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5132.028659] LustreError: 156564:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 5132.302525] Lustre: server umount lustre-MDT0000 complete [ 5149.237912] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 5162.145230] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 5201.622810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.859921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5201.892452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5201.962350] Lustre: lustre-MDT0000: new disk, initializing [ 5202.061272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5202.092300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5206.856471] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5218.172630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.285171] Lustre: 159873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5218.343425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5218.347039] Lustre: Skipped 1 previous similar message [ 5218.418374] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5218.440584] Lustre: lustre-MDT0001: new disk, initializing [ 5218.532246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5218.569224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5218.578490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5223.854850] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5233.752483] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5241.348188] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5249.501608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5249.786524] Lustre: lustre-OST0000: new disk, initializing [ 5249.793821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5249.804311] Lustre: 161841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5249.929456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.760361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5255.770558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5255.920640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5257.085327] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5266.437846] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5271.233577] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5273.084570] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:50:21 (1786420221) [ 5505.504447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5505.515820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5505.532427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.041130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.041234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.048446] Lustre: Skipped 1 previous similar message [ 5507.059100] Lustre: Skipped 1 previous similar message [ 5512.160878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5512.164373] Lustre: Skipped 2 previous similar messages [ 5517.279250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5517.281615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5517.294745] Lustre: Skipped 2 previous similar messages [ 5517.449430] Lustre: server umount lustre-MDT0000 complete [ 5520.807172] LustreError: 159864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420471 with bad export cookie 1679768824735182608 [ 5520.808689] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5520.819157] LustreError: 159864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5522.400318] LustreError: 159879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.400379] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.407949] LustreError: 159880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.408978] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5525.435784] Lustre: Skipped 1 previous similar message [ 5530.528616] LustreError: 159879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5535.199412] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5535.420266] Lustre: server umount lustre-MDT0001 complete [ 5545.959783] Lustre: server umount lustre-OST0000 complete [ 5556.784564] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 5567.046249] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 5605.375199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5605.693818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5605.752760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5605.926487] Lustre: lustre-MDT0000: new disk, initializing [ 5606.060496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5606.082389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5611.407771] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5621.857132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.004792] Lustre: 166503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5622.053504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5622.057591] Lustre: Skipped 1 previous similar message [ 5622.143577] Lustre: lustre-MDT0001: new disk, initializing [ 5622.229425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5622.300956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5622.322772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5627.150662] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5636.366509] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5642.399585] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5650.524798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5650.823700] Lustre: lustre-OST0000: new disk, initializing [ 5650.827798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5650.836257] Lustre: 168476:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5650.928957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5658.050970] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5660.697329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5660.708977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5660.798671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5669.738434] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5675.748598] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5676.010722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5681.253587] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5681.468403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5686.255375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5686.261378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5686.292255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5686.301365] Lustre: Skipped 2 previous similar messages [ 5689.748092] Lustre: server umount lustre-OST0000 complete [ 5696.482038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.510334] Lustre: Skipped 1 previous similar message [ 5700.536268] Lustre: server umount lustre-MDT0000 complete [ 5701.601708] LustreError: 166514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.619933] LustreError: 166514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5704.203713] LustreError: 166494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420654 with bad export cookie 1679768824735341438 [ 5704.208575] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5704.224494] LustreError: 166494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5704.623535] Lustre: server umount lustre-MDT0001 complete [ 5714.755793] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:57:43 (1786420663) [ 5731.984398] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 5745.501703] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 5789.709287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5789.857183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5789.877897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5789.931796] Lustre: lustre-MDT0000: new disk, initializing [ 5789.991657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5790.005607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5795.480868] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5806.613302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.721366] Lustre: 173234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5806.937292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5812.001523] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5822.058520] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5828.261636] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5835.031845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5835.415993] Lustre: lustre-OST0000: new disk, initializing [ 5835.418638] Lustre: Skipped 1 previous similar message [ 5835.421889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5835.425459] Lustre: Skipped 2 previous similar messages [ 5835.429961] Lustre: 175211:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5842.327068] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5843.572543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5843.607408] Lustre: Skipped 1 previous similar message [ 5843.628532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5843.765763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5852.108186] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5859.301195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5859.307934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.315049] LustreError: Skipped 1 previous similar message [ 5859.326589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.336973] Lustre: Skipped 2 previous similar messages [ 5859.342321] Lustre: Skipped 2 previous similar messages [ 5864.117285] Lustre: server umount lustre-OST0000 complete [ 5868.515382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5882.335127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5882.573458] Lustre: server umount lustre-MDT0000 complete [ 5883.882909] LustreError: 174401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.928383] LustreError: 174401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5887.054477] LustreError: 173228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420837 with bad export cookie 1679768824735342320 [ 5887.068349] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5887.080959] LustreError: 173228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5887.661580] Lustre: server umount lustre-MDT0001 complete [ 5898.722443] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:00:47 (1786420847) [ 5917.616707] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 5927.263434] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 5973.124563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.403646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5973.432939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5973.528620] Lustre: lustre-MDT0000: new disk, initializing [ 5973.597200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5973.602475] Lustre: Skipped 2 previous similar messages [ 5973.619536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5979.291135] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 5992.492911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5992.688298] Lustre: 179552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5993.156419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5999.837808] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6010.834449] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6017.608298] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6025.609867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6026.078722] Lustre: 181532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6033.075920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6033.262974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6034.953616] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6045.964644] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6053.866093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6053.895269] Lustre: Skipped 2 previous similar messages [ 6053.908633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6053.919804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6053.925351] Lustre: Skipped 9 previous similar messages [ 6067.167115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6067.399806] Lustre: server umount lustre-OST0000 complete [ 6077.737129] Lustre: server umount lustre-MDT0000 complete [ 6079.479085] LustreError: 179564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6079.542105] LustreError: 179564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6083.113485] LustreError: 180353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421033 with bad export cookie 1679768824735343405 [ 6083.130163] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6083.137262] LustreError: 180353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6083.565597] Lustre: server umount lustre-MDT0001 complete [ 6101.853362] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 6105.967653] Key type lgssc unregistered [ 6106.262322] LNet: 183413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6106.270755] LNetError: 183413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6106.283820] LNet: Removed LNI 192.168.204.150@tcp [ 6107.363602] Key type .llcrypt unregistered [ 6107.369237] Key type ._llcrypt unregistered [ 6142.402742] Key type ._llcrypt registered [ 6142.404989] Key type .llcrypt registered [ 6142.507857] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 6144.506335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6144.658274] alg: No test for adler32 (adler32-zlib) [ 6145.740834] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 6146.007758] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 6147.783170] Key type lgssc registered [ 6149.744768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6158.066221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6158.092804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.596163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6165.203924] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6170.094790] LustreError: 184836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.146619] LustreError: 184836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6175.202390] LustreError: 184837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6175.307362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.686140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6182.006469] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6191.240629] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6197.202312] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6203.799447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6204.100696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6205.137667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6211.019444] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6219.202587] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6241.248930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6241.257785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6241.283686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.278224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6242.292049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.306034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.329030] Lustre: Skipped 1 previous similar message [ 6244.207380] Lustre: server umount lustre-OST0000 complete [ 6252.515484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6252.519847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6252.528467] Lustre: Skipped 1 previous similar message [ 6252.532568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6252.535040] Lustre: Skipped 1 previous similar message [ 6257.632543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6257.651411] Lustre: Skipped 1 previous similar message [ 6261.729296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6261.738108] Lustre: Skipped 1 previous similar message [ 6263.263133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6263.369069] Lustre: server umount lustre-MDT0000 complete [ 6266.859181] LustreError: 184836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6266.899213] LustreError: 184836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6267.211418] LustreError: 184816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421217 with bad export cookie 13249181213201423425 [ 6267.212801] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.233618] LustreError: 184816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6267.546494] Lustre: server umount lustre-MDT0001 complete [ 6288.416178] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 6292.298950] Key type lgssc unregistered [ 6292.687346] LNet: 188420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6292.709484] LNetError: 188420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6292.729559] LNet: Removed LNI 192.168.204.150@tcp [ 6293.677292] Key type .llcrypt unregistered [ 6293.679059] Key type ._llcrypt unregistered [ 6328.036988] Key type ._llcrypt registered [ 6328.040303] Key type .llcrypt registered [ 6328.160675] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 6330.087949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6330.145570] alg: No test for adler32 (adler32-zlib) [ 6331.426936] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 6331.727265] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 6333.391167] Key type lgssc registered [ 6334.713650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6343.211833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6343.239985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6344.648694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6350.999961] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6354.912190] LustreError: 189837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.927882] LustreError: 189837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6359.581061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.082532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6364.634572] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6374.511548] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6381.389361] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6390.597106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.153573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6399.351778] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6409.744448] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6414.727909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6422.179991] Lustre: lustre-OST0000: Not available for connect from 192.168.204.50@tcp (stopping) [ 6425.061085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6425.067042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.073120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.102448] Lustre: Skipped 1 previous similar message [ 6427.257663] Lustre: lustre-OST0000: Not available for connect from 192.168.204.50@tcp (stopping) [ 6427.266837] Lustre: Skipped 1 previous similar message [ 6427.946303] Lustre: server umount lustre-OST0000 complete [ 6436.577644] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d308768a000.ost_server_uuid 50 [ 6438.246716] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6454.752725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6454.761311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.765305] Lustre: Skipped 1 previous similar message [ 6458.758023] Lustre: server umount lustre-MDT0000 complete [ 6459.879750] LustreError: 191271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.909847] LustreError: 191271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6463.375857] LustreError: 189817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421413 with bad export cookie 17686418404861501999 [ 6463.386214] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.389710] LustreError: 189817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6463.897325] Lustre: server umount lustre-MDT0001 complete [ 6475.028883] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:10:23 (1786421423) [ 6483.493039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.833956] LustreError: 193753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6483.895287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.060653] LustreError: 193754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6490.311438] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6494.189557] LustreError: 193753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6501.294975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.664172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6507.686935] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6517.374697] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6524.390150] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6533.319286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6533.989343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6543.708555] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6554.276467] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6554.523910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6562.892584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.265528] Lustre: lustre-OST0001: new disk, initializing [ 6563.268857] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6563.276908] Lustre: 196598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6563.417418] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6571.867882] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6572.252570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6572.265098] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6572.540813] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6582.842621] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6587.875057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.913192] Lustre: Skipped 1 previous similar message [ 6587.922348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6592.429244] Lustre: server umount lustre-OST0000 complete [ 6592.998899] LustreError: 195578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6593.032745] LustreError: 195578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6600.859761] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6601.073905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6607.070977] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6607.301775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6609.532171] LustreError: 195577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.548570] LustreError: 195577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6623.719139] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6623.725427] LustreError: Skipped 1 previous similar message [ 6623.731101] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.740759] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6623.764173] Lustre: Skipped 2 previous similar messages [ 6623.778824] Lustre: Skipped 2 previous similar messages [ 6628.647947] Lustre: server umount lustre-OST0001 complete [ 6633.961361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6633.993247] Lustre: Skipped 1 previous similar message [ 6639.992659] Lustre: server umount lustre-MDT0000 complete [ 6644.198117] LustreError: 193753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.230077] LustreError: 193753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6644.262947] LustreError: 193734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421594 with bad export cookie 17686418404861503070 [ 6644.263958] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6644.283252] LustreError: 193734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6644.736850] Lustre: server umount lustre-MDT0001 complete [ 6672.223484] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 00:13:40 (1786421620) [ 6680.430908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.843463] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6680.902554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6681.203977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6687.920668] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6697.368819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6697.448914] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6697.480971] Lustre: 200039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6703.551838] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6712.519910] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6719.851659] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6726.915209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6727.157100] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6732.992465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6734.458350] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6743.057556] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6751.126097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6751.239843] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6751.528532] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.539935] Lustre: Skipped 2 previous similar messages [ 6759.021384] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6769.509607] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6773.728458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6773.738694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6773.753678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6773.761119] Lustre: Skipped 4 previous similar messages [ 6778.871348] Lustre: server umount lustre-OST0000 complete [ 6780.548151] LustreError: 203360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.289187] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6797.291344] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.336368] LustreError: Skipped 2 previous similar messages [ 6797.342772] Lustre: Skipped 1 previous similar message [ 6801.763104] Lustre: server umount lustre-OST0001 complete [ 6807.528092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6807.544837] Lustre: Skipped 7 previous similar messages [ 6821.343598] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6821.688707] Lustre: server umount lustre-MDT0000 complete [ 6826.592386] LustreError: 201908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421777 with bad export cookie 17686418404861504470 [ 6826.607270] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6826.619178] LustreError: 201908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6827.061125] Lustre: server umount lustre-MDT0001 complete [ 6856.749844] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:16:45 (1786421805) [ 6875.319348] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 6885.172406] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 6925.336325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.678595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6925.707557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6925.773358] Lustre: lustre-MDT0000: new disk, initializing [ 6925.837484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6925.862661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6931.123467] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6941.983749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6942.109064] Lustre: 208137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6942.195126] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6942.199512] Lustre: Skipped 1 previous similar message [ 6942.262430] Lustre: lustre-MDT0001: new disk, initializing [ 6942.411821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6942.425498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6948.227017] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6959.483090] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6966.687752] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6976.616689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6977.104418] Lustre: lustre-OST0000: new disk, initializing [ 6977.107953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6977.116683] Lustre: 210108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6985.465061] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6987.446634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6987.455384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6987.733854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6996.746840] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7002.546654] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7003.048260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7008.796938] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7009.021927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7013.349037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.355277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7013.363525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7013.363533] Lustre: Skipped 4 previous similar messages [ 7013.372543] Lustre: Skipped 4 previous similar messages [ 7017.420392] Lustre: server umount lustre-OST0000 complete [ 7028.015571] Lustre: server umount lustre-MDT0000 complete [ 7028.706513] LustreError: 210116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.727690] LustreError: 210116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 7031.912590] LustreError: 208128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421982 with bad export cookie 17686418404861506038 [ 7031.920868] LustreError: 208128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7031.933767] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7032.341982] Lustre: server umount lustre-MDT0001 complete [ 7041.180148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7046.265367] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7055.222494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.834186] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7070.154742] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7077.270254] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7085.313461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7092.871384] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7101.372624] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7106.281832] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7106.617591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7110.908958] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7111.163854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7116.267594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7116.275284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7116.285030] Lustre: Skipped 3 previous similar messages [ 7119.306319] Lustre: server umount lustre-OST0000 complete [ 7126.170685] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7130.719710] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7160.019265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7165.942675] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7173.375802] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7178.589219] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 7180.905618] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7180.918161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7180.933487] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7180.954458] Lustre: Skipped 1 previous similar message [ 7186.405377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7186.407454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7186.418926] LustreError: Skipped 1 previous similar message [ 7186.434790] Lustre: Skipped 5 previous similar messages [ 7188.628456] Lustre: server umount lustre-OST0000 complete [ 7198.463908] Lustre: server umount lustre-MDT0000 complete [ 7202.463329] LustreError: 212306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422153 with bad export cookie 17686418404861506920 [ 7202.472703] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7202.476890] LustreError: 212306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7202.751614] Lustre: server umount lustre-MDT0001 complete [ 7211.421277] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:22:40 (1786422160) [ 7217.392384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.776660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7217.786457] Lustre: Skipped 6 previous similar messages [ 7222.219916] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7230.517675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.162530] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7243.747303] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7250.210536] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7258.939763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7267.490096] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7275.504156] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7281.894374] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7282.280718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7288.868854] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7289.223862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7297.212686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7297.422204] Lustre: lustre-OST0001: new disk, initializing [ 7297.425982] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7297.432877] Lustre: 221662:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7304.808898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7304.819860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7305.075504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7306.173937] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7316.351679] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7321.925603] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7322.128692] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7327.724669] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7327.929393] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7330.797403] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7330.799690] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7330.816740] LustreError: Skipped 2 previous similar messages [ 7330.848741] Lustre: Skipped 5 previous similar messages [ 7344.095648] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7344.184139] Lustre: server umount lustre-OST0001 complete [ 7346.149369] LustreError: 220115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.187297] LustreError: 220115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7351.734154] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7352.168635] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7357.795470] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7358.062872] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7388.142525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.787213] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7393.802133] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7393.810483] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7393.828145] Lustre: Skipped 1 previous similar message [ 7394.857311] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7403.137977] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7405.273350] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7409.125655] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7409.137813] LustreError: Skipped 1 previous similar message [ 7413.561549] Lustre: server umount lustre-OST0001 complete [ 7436.817345] Lustre: server umount lustre-OST0000 complete [ 7444.961980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7444.977864] Lustre: Skipped 17 previous similar messages [ 7447.051905] Lustre: server umount lustre-MDT0000 complete [ 7451.664357] LustreError: 218273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422402 with bad export cookie 17686418404861507886 [ 7451.669658] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7451.677609] LustreError: 218273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7458.209606] Lustre: server umount lustre-MDT0001 complete [ 7485.778720] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 00:27:14 (1786422434) [ 7493.457456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7493.704280] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7493.763577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7498.743709] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7506.868940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.998697] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7507.059771] Lustre: 226910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7512.082742] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7519.277696] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7526.014876] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7533.313352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.437988] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7540.064962] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7548.950790] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7559.421198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7559.500765] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7566.948886] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7576.333969] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7578.417182] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7583.779779] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7584.008477] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7588.295706] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7588.557273] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7594.780691] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d308aac9000.ost_server_uuid 50 [ 7596.360712] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d308aac9000.ost_server_uuid in IDLE state after 0 sec [ 7597.206311] Lustre: Permanently deactivating lustre-OST0001 [ 7597.218701] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7600.360240] Lustre: Permanently reactivating lustre-OST0001 [ 7605.222934] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7605.226885] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7605.245312] Lustre: Skipped 6 previous similar messages [ 7605.270152] LustreError: Skipped 4 previous similar messages [ 7609.884456] Lustre: server umount lustre-OST0001 complete [ 7619.858396] Lustre: server umount lustre-OST0000 complete [ 7630.140235] Lustre: server umount lustre-MDT0000 complete [ 7633.388759] LustreError: 226904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422583 with bad export cookie 17686418404861509167 [ 7633.394092] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7633.406462] LustreError: 226904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7633.706467] Lustre: server umount lustre-MDT0001 complete [ 7656.226419] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 00:30:04 (1786422604) [ 7664.557975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.786283] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7664.818370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7664.831492] Lustre: Skipped 5 previous similar messages [ 7669.627793] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7677.516165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.591305] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7677.612802] Lustre: 233607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7682.894170] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7690.188678] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7696.632370] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7704.712615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7704.980397] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7705.005523] Lustre: Permanently deactivating lustre-OST0000 [ 7705.023903] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7711.447174] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7713.599952] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7722.871825] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7731.420485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7731.507796] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7731.796747] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7731.802561] Lustre: Skipped 11 previous similar messages [ 7738.757189] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7741.516031] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7748.550821] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7750.678567] Lustre: Permanently reactivating lustre-OST0000 [ 7750.715210] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7750.725485] Lustre: Skipped 2 previous similar messages [ 7759.516101] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 48 seconds. I think it's dead, and I am evicting it. exp 00000000877d2e98, cur 1786422710 deadline 1786422692 last 1786422662 [ 7760.536209] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7760.566769] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7760.596789] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7760.739187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7768.775591] Lustre: server umount lustre-OST0001 complete [ 7779.511300] Lustre: server umount lustre-OST0000 complete [ 7789.415379] Lustre: server umount lustre-MDT0000 complete [ 7794.138028] LustreError: 233599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422744 with bad export cookie 17686418404861510854 [ 7794.142405] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7794.156554] LustreError: 233599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7794.653445] Lustre: server umount lustre-MDT0001 complete [ 7809.087625] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 7812.100916] Key type lgssc unregistered [ 7812.494150] LNet: 238471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7812.513219] LNetError: 238471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7812.539762] LNet: Removed LNI 192.168.204.150@tcp [ 7813.883932] Key type .llcrypt unregistered [ 7813.887193] Key type ._llcrypt unregistered [ 7845.229418] Key type ._llcrypt registered [ 7845.231427] Key type .llcrypt registered [ 7845.395533] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 7847.486310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7847.584741] alg: No test for adler32 (adler32-zlib) [ 7849.234419] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 7849.543321] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 7851.351193] Key type lgssc registered [ 7852.951931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7863.066165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7863.085445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7864.583327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7869.949906] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7875.052499] LustreError: 239939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7875.088805] LustreError: 239939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7878.804455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.994561] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7879.022539] Lustre: 239928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7879.033458] Lustre: Permanently deactivating lustre-MDT0001 [ 7879.045939] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7879.367428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7880.144966] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7880.160757] LustreError: 239970:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7880.188567] LustreError: 239970:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7880.218256] LustreError: 239970:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7884.919703] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7893.130496] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7899.800900] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7907.960342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7908.426751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.453571] LustreError: 241781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7910.477438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7915.522713] LustreError: 241781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.137124] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7920.632339] LustreError: 241781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7927.188633] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7928.818797] LustreError: 241783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.871947] LustreError: 241783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7935.517815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7935.725727] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7941.134577] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7941.147095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7943.451379] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7952.354665] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7956.424496] Lustre: Permanently reactivating lustre-MDT0001 [ 7956.448744] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7956.454620] Lustre: Skipped 2 previous similar messages [ 7962.282768] Lustre: lustre-MDT0001: Client 59d9bed1-5b1c-411f-ad95-a06cc62a9b4d (at 192.168.204.50@tcp) reconnecting [ 7970.285980] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7972.770515] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7979.205259] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7979.482821] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7985.367774] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7985.604400] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7988.486299] Lustre: Permanently deactivating lustre-MDT0001 [ 7988.502444] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7988.510311] Lustre: Skipped 2 previous similar messages [ 7995.051578] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8002.528179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8002.544922] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.555798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8002.587250] Lustre: Skipped 2 previous similar messages [ 8004.128762] Lustre: server umount lustre-MDT0000 complete [ 8007.660685] LustreError: 240685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8007.694115] LustreError: 240685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8008.257610] LustreError: 239920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422958 with bad export cookie 10871155900797582740 [ 8008.265330] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8008.273364] LustreError: 239920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8008.646020] Lustre: server umount lustre-MDT0001 complete [ 8015.514598] Lustre: server umount lustre-OST0000 complete [ 8029.931523] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:36:17 (1786422977) [ 8042.137830] Lustre: server umount lustre-OST0001 complete [ 8050.937390] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 8063.084777] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8103.206326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8103.607774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8103.619536] Lustre: Skipped 2 previous similar messages [ 8103.644145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8103.710938] Lustre: lustre-MDT0000: new disk, initializing [ 8103.774791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8103.792593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8109.242117] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8120.117802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.253830] Lustre: 248203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8120.324136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8120.328764] Lustre: Skipped 1 previous similar message [ 8120.397706] Lustre: lustre-MDT0001: new disk, initializing [ 8120.509319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8120.550657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8120.566101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8125.779571] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8135.749237] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8141.936464] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8149.624829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.017506] Lustre: lustre-OST0000: new disk, initializing [ 8150.024564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8150.037578] Lustre: 250173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8150.162571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8156.269215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8156.295558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8156.478785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8158.418424] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8168.740498] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8170.895099] LustreError: 248209:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8180.900404] LustreError: 248209:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8180.973644] LustreError: 249788:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8181.946148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8182.378898] Lustre: lustre-OST0001: new disk, initializing [ 8182.386971] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8182.400407] Lustre: 251289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8182.506234] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8189.407693] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8190.985113] LustreError: 249788:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8192.721500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8192.754036] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8193.010376] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8201.516162] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8204.202443] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8208.363565] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.374816] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8208.386641] Lustre: Skipped 1 previous similar message [ 8208.391543] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8208.391553] Lustre: Skipped 3 previous similar messages [ 8212.711309] Lustre: server umount lustre-OST0001 complete [ 8213.491397] LustreError: 250467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8213.513045] LustreError: 250467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8233.442990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8233.450924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8233.464774] Lustre: Skipped 1 previous similar message [ 8233.477797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8233.486221] Lustre: Skipped 2 previous similar messages [ 8239.077308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8239.084399] Lustre: Skipped 1 previous similar message [ 8244.191269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8244.204899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8244.238605] Lustre: Skipped 1 previous similar message [ 8244.502745] Lustre: server umount lustre-OST0000 complete [ 8253.921865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8253.930475] LustreError: Skipped 1 previous similar message [ 8253.939171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.956612] Lustre: Skipped 1 previous similar message [ 8253.963312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8256.346523] Lustre: server umount lustre-MDT0000 complete [ 8259.553674] LustreError: 248214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8259.583502] LustreError: 248214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 8260.390039] LustreError: 248194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423210 with bad export cookie 10871155900797585337 [ 8260.400937] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8260.414299] LustreError: 248194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8266.942672] Lustre: server umount lustre-MDT0001 complete [ 8280.987477] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 8284.489913] Key type lgssc unregistered [ 8284.878220] LNet: 253328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8284.887622] LNetError: 253328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8284.904128] LNet: Removed LNI 192.168.204.150@tcp [ 8285.899784] Key type .llcrypt unregistered [ 8285.901953] Key type ._llcrypt unregistered [ 8329.854800] Key type ._llcrypt registered [ 8329.858495] Key type .llcrypt registered [ 8329.975777] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8331.386540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8331.614179] alg: No test for adler32 (adler32-zlib) [ 8333.388822] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 8333.948994] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 8335.783313] Key type lgssc registered [ 8336.706827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8345.232338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8345.264229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.553874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8346.582763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8346.793875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8353.700416] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8362.913656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8362.980666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8363.002376] Lustre: 255542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8363.211711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8367.751318] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8376.505790] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8384.221523] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8392.506505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8392.758246] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8393.034746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8400.696511] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8402.221825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8409.568842] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8463.841777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.847531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8463.865470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8463.897347] Lustre: Skipped 1 previous similar message [ 8466.740561] Lustre: server umount lustre-OST0000 complete [ 8472.342742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8487.758890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8488.313632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.124847] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8503.671412] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8503.698261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8503.701277] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8503.743652] Lustre: Skipped 1 previous similar message [ 8505.410634] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8549.857756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8549.857905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8549.873260] LustreError: Skipped 1 previous similar message [ 8549.893386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8549.893395] Lustre: Skipped 1 previous similar message [ 8551.927675] Lustre: server umount lustre-OST0000 complete [ 8558.050958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8558.058966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8558.081569] Lustre: Skipped 1 previous similar message [ 8558.091969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8558.101517] Lustre: Skipped 1 previous similar message [ 8560.099250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8560.103410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.725856] Lustre: server umount lustre-MDT0000 complete [ 8567.168407] LustreError: 255533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423517 with bad export cookie 12601002169511859226 [ 8567.173513] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8567.180575] LustreError: 255533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8567.655465] Lustre: server umount lustre-MDT0001 complete [ 8581.262679] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 8584.156920] Key type lgssc unregistered [ 8584.452931] LNet: 260967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8584.464948] LNetError: 260967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8584.488234] LNet: Removed LNI 192.168.204.150@tcp [ 8585.605354] Key type .llcrypt unregistered [ 8585.607309] Key type ._llcrypt unregistered [ 8617.109849] Key type ._llcrypt registered [ 8617.113659] Key type .llcrypt registered [ 8617.288671] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8618.543774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8618.679311] alg: No test for adler32 (adler32-zlib) [ 8619.830696] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 8620.131360] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 8621.791383] Key type lgssc registered [ 8623.476584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8632.844380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8632.865825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8634.364652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8639.479961] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8643.575193] LustreError: 262387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8643.619556] LustreError: 262387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8648.684198] LustreError: 262388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8650.689482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.328598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8658.231450] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8667.476355] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8674.972971] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8684.629421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8684.986751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8692.939430] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8703.478182] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8704.460923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8735.201682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8735.219961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8735.227867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8735.233927] Lustre: Skipped 1 previous similar message [ 8735.242953] Lustre: Skipped 1 previous similar message [ 8740.366779] Lustre: server umount lustre-OST0000 complete [ 8745.447283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8745.460407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8745.478521] Lustre: Skipped 1 previous similar message [ 8745.504515] Lustre: Skipped 1 previous similar message [ 8750.561127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8750.571994] Lustre: Skipped 1 previous similar message [ 8751.071150] Lustre: server umount lustre-MDT0000 complete [ 8754.661205] LustreError: 262388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.702548] LustreError: 262388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8755.985527] LustreError: 262369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423706 with bad export cookie 3981777240501873680 [ 8756.014739] LustreError: 262369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8756.020966] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8756.420497] Lustre: server umount lustre-MDT0001 complete [ 8772.598823] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 8776.228631] Key type lgssc unregistered [ 8776.696931] LNet: 266697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8776.707514] LNetError: 266697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8776.726284] LNet: Removed LNI 192.168.204.150@tcp [ 8777.958155] Key type .llcrypt unregistered [ 8777.961147] Key type ._llcrypt unregistered [ 8800.861974] Key type ._llcrypt registered [ 8800.869156] Key type .llcrypt registered [ 8801.033788] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8802.959136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8803.051623] alg: No test for adler32 (adler32-zlib) [ 8804.195813] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 8804.589789] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 8806.295250] Key type lgssc registered [ 8807.710876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8818.312128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8818.344652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.940290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8825.960714] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8830.436331] LustreError: 267918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8830.476253] LustreError: 267918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8835.557459] LustreError: 267919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8836.798134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.336757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8842.134833] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8849.686708] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8857.400952] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8865.881679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8866.272313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8875.028565] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8883.347252] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8889.852893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8905.184189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8905.189839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8905.207285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8905.224546] Lustre: Skipped 1 previous similar message [ 8907.222703] Lustre: server umount lustre-OST0000 complete [ 8913.890643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8913.901595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.919374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8913.927195] Lustre: Skipped 1 previous similar message [ 8915.426869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.429074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8915.449265] Lustre: Skipped 1 previous similar message [ 8917.475313] Lustre: server umount lustre-MDT0000 complete [ 8920.547219] LustreError: 267914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8920.578412] LustreError: 267914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8921.239462] LustreError: 267897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423871 with bad export cookie 7045336185236168440 [ 8921.257326] LustreError: 267897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8921.257388] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8921.593612] Lustre: server umount lustre-MDT0001 complete [ 8935.295888] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 8939.334480] Key type lgssc unregistered [ 8939.772026] LNet: 271623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8939.781457] LNetError: 271623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8939.801754] LNet: Removed LNI 192.168.204.150@tcp [ 8940.921727] Key type .llcrypt unregistered [ 8940.923616] Key type ._llcrypt unregistered [ 8973.424622] Key type ._llcrypt registered [ 8973.428587] Key type .llcrypt registered [ 8973.576622] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8974.810384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8974.836215] alg: No test for adler32 (adler32-zlib) [ 8976.512209] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 8976.877502] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 8978.631160] Key type lgssc registered [ 8980.205242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8988.876403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8988.904852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.363601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8995.474033] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8999.921181] LustreError: 273043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.937599] LustreError: 273043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9005.028607] LustreError: 273044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9005.137044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9005.412135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9011.431258] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9020.722159] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9028.791330] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9036.410072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9036.657400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9044.091725] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9052.510258] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9060.231182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9091.043220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9091.057335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9091.083438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9091.099888] Lustre: Skipped 1 previous similar message [ 9093.670476] Lustre: server umount lustre-OST0000 complete [ 9101.284948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9101.301453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9101.323443] Lustre: Skipped 2 previous similar messages [ 9104.615737] Lustre: server umount lustre-MDT0000 complete [ 9106.402789] LustreError: 275538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9106.455157] LustreError: 275538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9108.575035] LustreError: 273023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424059 with bad export cookie 15418419888100203181 [ 9108.583796] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9108.590940] LustreError: 273023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9108.917787] Lustre: server umount lustre-MDT0001 complete [ 9122.089775] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 9125.558043] Key type lgssc unregistered [ 9126.017623] LNet: 277376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9126.026428] LNetError: 277376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9126.043283] LNet: Removed LNI 192.168.204.150@tcp [ 9127.177793] Key type .llcrypt unregistered [ 9127.179803] Key type ._llcrypt unregistered [ 9149.639433] Key type ._llcrypt registered [ 9149.641892] Key type .llcrypt registered [ 9149.752422] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9151.710824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9151.867815] alg: No test for adler32 (adler32-zlib) [ 9152.880411] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 9153.137360] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 9154.863842] Key type lgssc registered [ 9156.819923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9167.022591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9167.051717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.698776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9174.253437] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9179.108770] LustreError: 278614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.159817] LustreError: 278614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9184.227493] LustreError: 278616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9185.033046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.621071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9191.650239] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9200.268864] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9207.733302] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9215.407494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9215.808621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9222.564591] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9231.783428] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9239.349163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9251.808602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9251.815401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9251.827085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9254.881691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9254.890704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9254.914527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9257.967153] Lustre: server umount lustre-OST0000 complete [ 9262.560591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9262.569771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9262.589146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9262.595715] Lustre: Skipped 1 previous similar message [ 9265.058041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9265.066893] Lustre: Skipped 3 previous similar messages [ 9267.548306] Lustre: server umount lustre-MDT0000 complete [ 9270.246659] LustreError: 278599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9270.271768] LustreError: 278599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9271.227343] LustreError: 278582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424221 with bad export cookie 17398046422925515191 [ 9271.230470] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9271.251755] LustreError: 278582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9271.610713] Lustre: server umount lustre-MDT0001 complete [ 9283.776738] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 9286.898740] Key type lgssc unregistered [ 9287.183535] LNet: 282341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9287.192317] LNetError: 282341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9287.213860] LNet: Removed LNI 192.168.204.150@tcp [ 9288.069940] Key type .llcrypt unregistered [ 9288.072475] Key type ._llcrypt unregistered [ 9302.961362] Key type ._llcrypt registered [ 9302.972351] Key type .llcrypt registered [ 9303.150552] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9321.835846] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 9339.056776] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9341.065406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9341.103431] alg: No test for adler32 (adler32-zlib) [ 9342.300441] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 9342.657265] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 9344.359197] Key type lgssc registered [ 9345.800973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9382.454568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9382.480031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9383.679257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9383.717098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9383.852261] Lustre: lustre-MDT0000: new disk, initializing [ 9383.925633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9383.954470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9389.674799] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9401.647392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9401.798374] Lustre: 285895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9401.870264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9401.874814] Lustre: Skipped 1 previous similar message [ 9401.946665] Lustre: lustre-MDT0001: new disk, initializing [ 9402.062834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9402.097706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9402.121333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9407.051916] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9416.240274] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9422.160125] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9428.643039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9429.008788] Lustre: lustre-OST0000: new disk, initializing [ 9429.016811] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9429.026933] Lustre: 287867:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9429.132891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9435.046996] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9435.742295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9435.771450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9436.039599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9444.900123] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9449.636238] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9449.915522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9454.673302] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9454.964728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9461.730808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.737786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9461.738788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9461.755685] Lustre: Skipped 1 previous similar message [ 9463.568548] Lustre: server umount lustre-OST0000 complete [ 9468.903976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9468.919459] LustreError: Skipped 1 previous similar message [ 9468.930820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9468.961442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9468.971713] Lustre: Skipped 1 previous similar message [ 9471.979791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9471.997856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9472.023968] Lustre: Skipped 1 previous similar message [ 9473.848549] Lustre: server umount lustre-MDT0000 complete [ 9478.099890] LustreError: 285889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424428 with bad export cookie 13994163819421166508 [ 9478.111680] LustreError: 285889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9478.113296] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9478.531971] Lustre: server umount lustre-MDT0001 complete [ 9488.195617] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:00:36 (1786424436) [ 9495.662905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9496.040679] LustreError: 290279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9496.064159] LustreError: 290279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9496.160340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9501.164481] LustreError: 290278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9501.195831] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9506.276345] LustreError: 290279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.370910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.119728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9517.072336] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9525.270443] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9531.696823] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9538.812069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9539.143667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9546.191204] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9554.285637] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9566.183635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9566.193882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9566.197196] Lustre: Skipped 1 previous similar message [ 9566.209603] Lustre: Skipped 1 previous similar message [ 9571.274368] Lustre: server umount lustre-OST0000 complete [ 9576.423873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9576.427915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9576.455032] Lustre: Skipped 1 previous similar message [ 9576.485100] Lustre: Skipped 1 previous similar message [ 9581.537313] LustreError: 290273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9581.573271] LustreError: 290273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9581.595279] Lustre: server umount lustre-MDT0000 complete [ 9585.640589] LustreError: 291235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9585.668577] LustreError: 291235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9585.679416] LustreError: 290258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424536 with bad export cookie 13994163819421167390 [ 9585.685824] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9585.691384] LustreError: 290258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9586.200134] Lustre: server umount lustre-MDT0001 complete [ 9600.030110] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 9602.747598] Key type lgssc unregistered [ 9603.128924] LNet: 293832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9603.136723] LNetError: 293832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9603.162542] LNet: Removed LNI 192.168.204.150@tcp [ 9603.992337] Key type .llcrypt unregistered [ 9603.995256] Key type ._llcrypt unregistered [ 9634.627555] Key type ._llcrypt registered [ 9634.630389] Key type .llcrypt registered [ 9634.718442] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 9653.930190] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9655.401214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9655.477932] alg: No test for adler32 (adler32-zlib) [ 9656.642782] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 9656.884571] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 9658.591924] Key type lgssc registered [ 9659.978981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9707.636867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9707.681058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9708.930961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9708.961960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9709.027568] Lustre: lustre-MDT0000: new disk, initializing [ 9709.132424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9709.169617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9713.768424] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9724.921793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.164523] Lustre: 297661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9725.297722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9725.309382] Lustre: Skipped 1 previous similar message [ 9725.463567] Lustre: lustre-MDT0001: new disk, initializing [ 9725.599636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9725.719639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9725.746951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9730.914555] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9740.281931] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9746.352371] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9754.272680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9754.618759] Lustre: lustre-OST2710: new disk, initializing [ 9754.626065] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9754.636491] Lustre: 299628:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9754.745788] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9762.892671] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9764.986976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9765.013554] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9765.230079] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9773.280170] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9780.585924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9780.763482] Lustre: lustre-OST03e8: new disk, initializing [ 9780.767509] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9780.773546] Lustre: 300695:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9780.910903] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9788.690490] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9789.179927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9789.214771] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9789.434478] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9798.152482] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9806.097934] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9806.450380] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9812.818735] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9813.085154] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9818.891768] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9819.156771] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9824.180175] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9824.403555] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9830.377504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.380591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9830.388241] Lustre: Skipped 3 previous similar messages [ 9830.402812] Lustre: Skipped 3 previous similar messages [ 9830.850468] Lustre: server umount lustre-MDT0000 complete [ 9834.794764] LustreError: 297653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424785 with bad export cookie 11819272060179152710 [ 9834.816124] LustreError: 297653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9834.822560] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9835.230024] Lustre: server umount lustre-MDT0001 complete [ 9849.977310] Lustre: server umount lustre-OST2710 complete [ 9851.938698] Lustre: 295457:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786424786/real 1786424786] req@000000002fb0ff65 x1873201968780800/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786424802 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9851.986958] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9853.688715] Lustre: server umount lustre-OST03e8 complete [ 9872.520417] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 9883.511274] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9923.821191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.138600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9924.172402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9924.303363] Lustre: lustre-MDT0000: new disk, initializing [ 9924.385301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9924.406736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9930.726659] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9942.949544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.143512] Lustre: 305946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9943.291198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9943.304184] Lustre: Skipped 1 previous similar message [ 9943.457383] Lustre: lustre-MDT0001: new disk, initializing [ 9943.615580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9943.675535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9943.692087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9948.931595] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9961.055550] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9966.960524] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9974.430635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9974.682239] Lustre: lustre-OST0000: new disk, initializing [ 9974.686340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9974.693949] Lustre: 307914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9974.887935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9980.536562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9980.552102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9980.713974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9981.873747] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9993.431878] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9998.866471] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9999.230790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10004.857537] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10005.114294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10011.623399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10011.624658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10011.627043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10011.643967] LustreError: Skipped 1 previous similar message [10011.656108] Lustre: Skipped 1 previous similar message [10016.739819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10016.746042] Lustre: Skipped 2 previous similar messages [10021.343499] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10021.488037] Lustre: server umount lustre-OST0000 complete [10025.442298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10025.449587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.462955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10031.784520] Lustre: server umount lustre-MDT0000 complete [10035.586844] LustreError: 305938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424986 with bad export cookie 11819272060179154390 [10035.589968] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10035.596324] LustreError: 305938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10035.904541] Lustre: server umount lustre-MDT0001 complete [10045.793265] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:09:54 (1786424994) [10047.689269] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10049.606574] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:09:58 (1786424998) [10052.903283] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10085.242763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10085.614594] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10085.633945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10085.751098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10092.583203] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10103.071351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.155845] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10103.186899] Lustre: 311775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10107.640784] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10129.127615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10129.332249] Lustre: Denying initial registration attempt from nid 192.168.204.150@tcp, specified as failover [10129.347139] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10129.359173] LustreError: 313229:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10129.366927] LustreError: 313229:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10129.378626] LustreError: 313229:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10129.443348] Lustre: server umount lustre-OST0000 complete [10129.446409] LustreError: 313229:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [10139.105199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10139.105270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10139.117829] Lustre: Skipped 1 previous similar message [10139.131153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10139.139105] Lustre: Skipped 3 previous similar messages [10149.346216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10149.348940] Lustre: Skipped 2 previous similar messages [10150.367157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10150.490638] Lustre: server umount lustre-MDT0000 complete [10154.063168] LustreError: 311767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425104 with bad export cookie 11819272060179155272 [10154.090065] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10154.316967] Lustre: server umount lustre-MDT0001 complete [10164.955801] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [10168.027845] Key type lgssc unregistered [10168.329457] LNet: 314355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10168.336938] LNetError: 314355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10168.357667] LNet: Removed LNI 192.168.204.150@tcp [10169.495405] Key type .llcrypt unregistered [10169.497569] Key type ._llcrypt unregistered [10181.918588] Key type ._llcrypt registered [10181.922645] Key type .llcrypt registered [10182.077151] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10182.918945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10182.955794] alg: No test for adler32 (adler32-zlib) [10184.002943] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [10184.201262] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [10185.863467] Key type lgssc registered [10187.009750] Lustre: Echo OBD driver; http://www.lustre.org/ [10239.966938] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10251.375619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10251.397885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10252.738987] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10252.760763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10252.886985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10259.406954] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10269.761471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10269.893118] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10269.929516] Lustre: 317121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10270.236380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10275.106502] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10283.597323] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10289.012598] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10308.744166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10309.032788] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10309.223238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10317.045541] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10324.920303] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10327.015852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10327.026687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.053413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10327.520437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10327.533285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.559406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10332.641613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10332.652134] Lustre: Skipped 1 previous similar message [10337.764795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10337.776494] Lustre: Skipped 1 previous similar message [10341.344241] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10341.485697] Lustre: server umount lustre-OST0000 complete [10346.981162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10346.987896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10347.003718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.578933] Lustre: server umount lustre-MDT0000 complete [10356.889867] LustreError: 317113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425307 with bad export cookie 5069247609270201519 [10356.901232] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10357.159705] Lustre: server umount lustre-MDT0001 complete [10369.252499] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [10372.557265] Key type lgssc unregistered [10372.842249] LNet: 320698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10372.856543] LNetError: 320698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10373.926750] LNet: Removed LNI 192.168.204.150@tcp [10375.010594] Key type .llcrypt unregistered [10375.018617] Key type ._llcrypt unregistered [10407.157715] Key type ._llcrypt registered [10407.162640] Key type .llcrypt registered [10407.388592] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10408.962898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10409.000704] alg: No test for adler32 (adler32-zlib) [10410.247097] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [10410.530157] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [10412.239170] Key type lgssc registered [10414.526368] Lustre: Echo OBD driver; http://www.lustre.org/ [10424.099389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10424.121618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.495387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10430.671364] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10436.071917] LustreError: 322121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.089583] LustreError: 322121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10441.186326] LustreError: 322122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10441.519904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10442.024359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10448.381698] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10456.761324] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10463.318694] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10473.519836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10473.814913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10480.693854] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10489.400573] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10508.769923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10508.777910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.792988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10511.512477] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.50@tcp (stopping) [10511.850477] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10511.873140] Lustre: Skipped 1 previous similar message [10514.144258] Lustre: server umount lustre-MDT0000 complete [10516.605665] LustreError: 322118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10518.490540] LustreError: 322102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425469 with bad export cookie 10820315588182504762 [10518.504526] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10518.511148] LustreError: 322102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10521.729186] LustreError: 322855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10521.753742] LustreError: 322855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10521.757441] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.50@tcp (stopping) [10521.767848] Lustre: Skipped 3 previous similar messages [10522.082972] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.906657] Lustre: server umount lustre-MDT0001 complete [10531.431688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10541.651715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10543.521295] LustreError: 321429:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000090446398 x1873202759382784/t0(0) o250->MGC192.168.204.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10543.924256] LustreError: 323952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10543.964145] LustreError: 323952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10544.119922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10544.239531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10550.077862] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10553.345810] LustreError: 325737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10553.359412] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10553.389304] LustreError: 325737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10560.495245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.901126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10565.247658] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10566.135903] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10573.490480] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10574.941248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10580.629696] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10582.455206] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10590.181441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10590.191756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10590.223022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10590.245247] Lustre: Skipped 2 previous similar messages [10591.716705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10593.972527] Lustre: server umount lustre-OST0000 complete [10601.955761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10601.956220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10601.959633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10601.959643] Lustre: Skipped 2 previous similar messages [10602.007248] Lustre: Skipped 2 previous similar messages [10612.191188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10612.196673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10612.211706] Lustre: Skipped 4 previous similar messages [10612.445399] Lustre: server umount lustre-MDT0000 complete [10616.525122] LustreError: 322100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425567 with bad export cookie 10820315588182512448 [10616.544677] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10616.548064] LustreError: 322100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10616.929354] Lustre: server umount lustre-MDT0001 complete [10629.181850] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [10632.558753] Key type lgssc unregistered [10632.951600] LNet: 328420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10632.964354] LNetError: 328420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10632.986598] LNet: Removed LNI 192.168.204.150@tcp [10634.061802] Key type .llcrypt unregistered [10634.064700] Key type ._llcrypt unregistered [10663.736854] Key type ._llcrypt registered [10663.742956] Key type .llcrypt registered [10664.009429] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10666.176111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10666.384502] alg: No test for adler32 (adler32-zlib) [10667.498758] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [10667.948532] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [10669.735238] Key type lgssc registered [10671.447595] Lustre: Echo OBD driver; http://www.lustre.org/ [10680.690848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10680.712249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.426635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10688.391610] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10691.556095] LustreError: 329839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.590048] LustreError: 329839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10696.673530] LustreError: 329838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10699.157593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10699.485782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10703.887678] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10712.163250] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10719.080435] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10723.303309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10723.327038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10723.335797] Lustre: Skipped 1 previous similar message [10724.387199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10724.394027] Lustre: Skipped 1 previous similar message [10729.446245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10729.459716] Lustre: Skipped 1 previous similar message [10729.692717] Lustre: server umount lustre-MDT0000 complete [10734.092057] LustreError: 329820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425684 with bad export cookie 6814853071608945899 [10734.096724] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10734.106301] LustreError: 329820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10734.499865] Lustre: server umount lustre-MDT0001 complete [10750.955323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.323446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10751.371559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10751.638936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10758.063727] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10767.699665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10767.832446] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10767.851284] Lustre: 332428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10768.120734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10772.738696] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10780.849932] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10787.649465] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10798.764940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10799.038375] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10799.047192] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10799.060331] LustreError: 334313:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10799.064541] LustreError: 334313:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10799.068662] LustreError: 334313:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10799.140610] Lustre: server umount lustre-OST0000 complete [10799.144953] LustreError: 334313:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10811.065510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10811.224905] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10811.419489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10818.271062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10818.626849] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10827.227928] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10837.707856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10837.941580] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10838.149779] Lustre: lustre-OST0001: new disk, initializing [10838.159984] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10838.172693] Lustre: 335854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10838.381298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10845.412152] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10846.480536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10846.515036] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10846.806227] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10853.947712] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10862.056354] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10862.077311] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10862.087891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10862.100246] Lustre: Skipped 2 previous similar messages [10862.131026] Lustre: Skipped 1 previous similar message [10863.372869] Lustre: server umount lustre-OST0001 complete [10867.173343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10867.176493] LustreError: 334781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.176561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10867.190829] Lustre: Skipped 1 previous similar message [10867.209782] LustreError: 334781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10872.294541] LustreError: 336475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.324763] LustreError: 336475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10873.258685] Lustre: server umount lustre-OST0000 complete [10877.412628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10877.426939] Lustre: Skipped 1 previous similar message [10877.429787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10877.441394] Lustre: Skipped 3 previous similar messages [10883.693118] Lustre: server umount lustre-MDT0000 complete [10887.649280] LustreError: 332434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10887.659073] LustreError: 332434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10887.833866] LustreError: 332421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425838 with bad export cookie 6814853071608946452 [10887.845042] LustreError: 332421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10887.846182] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10888.231187] Lustre: server umount lustre-MDT0001 complete [10900.087857] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [10903.885894] Key type lgssc unregistered [10904.318258] LNet: 337816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10904.332928] LNetError: 337816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10904.366714] LNet: Removed LNI 192.168.204.150@tcp [10905.724622] Key type .llcrypt unregistered [10905.729665] Key type ._llcrypt unregistered [10987.214253] Key type ._llcrypt registered [10987.218757] Key type .llcrypt registered [10987.464479] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [11005.701755] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11007.434170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11007.682508] alg: No test for adler32 (adler32-zlib) [11009.435553] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [11009.668601] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [11011.399209] Key type lgssc registered [11013.734144] Lustre: Echo OBD driver; http://www.lustre.org/ [11052.432391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11052.506199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11053.747673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11053.772524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11053.846555] Lustre: lustre-MDT0000: new disk, initializing [11053.912136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11053.933714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11058.152591] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11070.258748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11070.398500] Lustre: 342755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11070.475980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11070.479508] Lustre: Skipped 1 previous similar message [11070.551371] Lustre: lustre-MDT0001: new disk, initializing [11070.695745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11070.783041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11070.813552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11076.314565] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11086.784779] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11093.127761] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11100.831989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11101.239802] Lustre: lustre-OST0000: new disk, initializing [11101.244788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11101.262701] Lustre: 344727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11101.360346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11108.254456] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11109.972099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11109.993196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11110.186275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11117.948115] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11122.214896] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11122.417983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11127.009626] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11127.245758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11130.850786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11130.854302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11130.859844] LustreError: Skipped 1 previous similar message [11130.862451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11130.869978] Lustre: Skipped 1 previous similar message [11135.423300] Lustre: server umount lustre-OST0000 complete [11141.090652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11141.096355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11141.112609] Lustre: Skipped 1 previous similar message [11141.131800] Lustre: Skipped 2 previous similar messages [11146.210710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11146.224336] Lustre: Skipped 1 previous similar message [11151.334372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11151.339388] Lustre: Skipped 1 previous similar message [11154.402310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11154.655575] Lustre: server umount lustre-MDT0000 complete [11158.846381] LustreError: 342748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426109 with bad export cookie 4436645665100705253 [11158.850304] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11158.861472] LustreError: 342748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11159.381978] Lustre: server umount lustre-MDT0001 complete [11170.447386] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 01:28:38 (1786426118) [11179.691409] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 01:28:47 (1786426127) [11189.004455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11189.374699] LustreError: 347482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11189.414621] LustreError: 347482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11189.514071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11194.861225] LustreError: 347483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11195.969318] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11198.963691] LustreError: 347482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11204.078798] LustreError: 347483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11207.328804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11207.726111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11214.240666] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11222.687848] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11228.659425] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11234.786894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11235.072407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11241.757782] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11251.058835] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11271.053260] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11273.407545] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11275.785306] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11278.429777] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11281.152297] Lustre: Failing over lustre-MDT0000 [11281.528441] Lustre: server umount lustre-MDT0000 complete [11283.598143] LustreError: 347478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11284.454254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11284.464165] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11284.490209] Lustre: Skipped 1 previous similar message [11293.823905] LustreError: 347477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.851666] LustreError: 347477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11300.833502] Lustre: 340981:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786426235/real 1786426235] req@0000000029e7a376 x1873203387510400/t0(0) o400->MGC192.168.204.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786426251 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11300.886195] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11302.589588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11311.080916] LustreError: 347477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.085409] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3d9221b9de9ab571 [11311.101153] LustreError: 347477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11311.121117] Lustre: MGC192.168.204.150@tcp: Connection restored to 0@lo (at 0@lo) [11311.448291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11311.528058] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11314.312903] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11316.740785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11316.781724] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11316.852939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11317.018781] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11328.059035] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11330.247592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11335.358480] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11337.585543] Lustre: Failing over lustre-MDT0000 [11337.809867] Lustre: server umount lustre-MDT0000 complete [11342.304551] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11342.347171] Lustre: Skipped 2 previous similar messages [11345.031870] LustreError: 347477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11345.052306] LustreError: 347477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11357.671515] Lustre: 340979:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786426292/real 1786426292] req@0000000069555f30 x1873203387532928/t0(0) o400->MGC192.168.204.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786426308 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11357.723178] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11358.924361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11367.908950] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3d9221b9de9ab888 [11367.927934] Lustre: MGC192.168.204.150@tcp: Connection restored to 0@lo (at 0@lo) [11367.945502] Lustre: Skipped 2 previous similar messages [11368.256962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11368.320982] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11371.651522] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11373.579444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11373.678893] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11373.816052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11374.489536] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11384.870743] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11386.270815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11394.021993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11394.024139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11394.039934] Lustre: Skipped 3 previous similar messages [11394.056736] Lustre: Skipped 2 previous similar messages [11398.149357] Lustre: server umount lustre-MDT0000 complete [11402.651696] LustreError: 347463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426353 with bad export cookie 4436645665100707976 [11402.658923] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11402.666190] LustreError: 347463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11403.029623] Lustre: server umount lustre-MDT0001 complete [11416.827949] Lustre: server umount lustre-OST0000 complete [11434.614436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11434.877633] LustreError: 353850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11434.893519] LustreError: 353850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [11434.938557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11440.663675] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11451.185723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11457.421252] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11466.348807] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11473.129428] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11480.321332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11487.749162] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11495.601829] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11505.291867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11512.800945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11512.811130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.830704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11515.883040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11517.734766] Lustre: server umount lustre-OST0000 complete [11523.552340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11531.233030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11531.245232] Lustre: Skipped 6 previous similar messages [11536.351180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11536.619629] Lustre: server umount lustre-MDT0000 complete [11540.757956] LustreError: 353832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426491 with bad export cookie 4436645665100708718 [11540.766106] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11540.772248] LustreError: 353832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11541.307886] Lustre: server umount lustre-MDT0001 complete [11556.810324] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [11560.673613] Key type lgssc unregistered [11561.027645] LNet: 357409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11561.036389] LNetError: 357409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11561.053473] LNet: Removed LNI 192.168.204.150@tcp [11561.932513] Key type .llcrypt unregistered [11561.935097] Key type ._llcrypt unregistered [11594.479144] Key type ._llcrypt registered [11594.481364] Key type .llcrypt registered [11594.638479] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11595.700137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11595.752260] alg: No test for adler32 (adler32-zlib) [11597.136802] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [11597.425100] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [11599.175909] Key type lgssc registered [11600.378802] Lustre: Echo OBD driver; http://www.lustre.org/ [11608.905595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11608.936217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.628577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11616.652691] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11620.842149] LustreError: 358928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11620.863264] LustreError: 358928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11625.955495] LustreError: 358927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11627.448410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11627.741246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11633.442526] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11641.352832] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11647.790672] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11655.642126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11656.082222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11664.617887] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11674.160836] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11680.732471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11695.856114] Lustre: server umount lustre-MDT0000 complete [11696.112432] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11696.146757] Lustre: Skipped 1 previous similar message [11696.162196] LustreError: 358927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11696.212676] LustreError: 358927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11701.221969] LustreError: 358928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11701.244086] LustreError: 358928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11701.335733] LustreError: 358909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426651 with bad export cookie 6939896400078265454 [11701.363214] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11701.703673] Lustre: server umount lustre-MDT0001 complete [11717.634492] Lustre: server umount lustre-OST0000 complete [11735.027803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11735.584518] LustreError: 362667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11735.662944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11741.652364] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11746.277654] LustreError: 362668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11746.310937] LustreError: 362668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11751.916671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11752.271307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11757.944448] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11767.063791] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11773.870952] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11783.279590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11783.588100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11792.056651] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11801.133711] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11806.196745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11818.977332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11818.988294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11819.003313] Lustre: Skipped 1 previous similar message [11819.006121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11821.540809] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11821.541386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11821.580950] Lustre: Skipped 2 previous similar messages [11825.190281] Lustre: server umount lustre-MDT0000 complete [11826.660984] LustreError: 365158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11826.708146] LustreError: 365158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11829.910178] LustreError: 362647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426780 with bad export cookie 6939896400078266560 [11829.911558] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11829.941913] LustreError: 362647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11830.401904] Lustre: server umount lustre-MDT0001 complete [11844.820719] Lustre: server umount lustre-OST0000 complete [11865.383835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11865.966473] LustreError: 366582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.096843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11872.243926] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11882.707613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.931844] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11897.823398] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11904.138340] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11911.908181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11912.243837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11912.248583] Lustre: Skipped 1 previous similar message [11919.022971] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11926.640569] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11935.806315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11945.953328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11945.972255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.985282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11945.999216] Lustre: Skipped 2 previous similar messages [11946.016782] Lustre: Skipped 1 previous similar message [11949.599171] Lustre: server umount lustre-OST0000 complete [11956.193061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.201523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11956.215599] Lustre: Skipped 1 previous similar message [11956.238113] Lustre: Skipped 1 previous similar message [11961.315406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11961.323015] Lustre: Skipped 1 previous similar message [11961.607899] Lustre: server umount lustre-MDT0000 complete [11966.203540] LustreError: 366563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426916 with bad export cookie 6939896400078267589 [11966.219601] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11966.669774] Lustre: server umount lustre-MDT0001 complete [11982.941838] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [11987.645588] Key type lgssc unregistered [11988.182449] LNet: 370144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11988.203308] LNetError: 370144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11988.217429] LNet: Removed LNI 192.168.204.150@tcp [11989.549405] Key type .llcrypt unregistered [11989.555513] Key type ._llcrypt unregistered [12023.867462] Key type ._llcrypt registered [12023.870341] Key type .llcrypt registered [12023.988816] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12025.299337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12025.342071] alg: No test for adler32 (adler32-zlib) [12026.459995] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [12026.710182] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [12028.425076] Key type lgssc registered [12029.764125] Lustre: Echo OBD driver; http://www.lustre.org/ [12039.675509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12039.694881] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12039.706389] LustreError: 371594:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12039.729975] LustreError: 371594:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12039.734434] LustreError: 371594:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12039.739508] LustreError: 371594:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12039.744505] LustreError: 371594:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [12047.045520] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12047.053865] LustreError: 371992:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12047.068882] LustreError: 371992:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12047.078684] LustreError: 371992:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12047.084544] LustreError: 371992:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12047.091932] LustreError: 371992:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [12069.961897] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12073.433519] Key type lgssc unregistered [12073.867716] LNet: 372998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12073.876141] LNetError: 372998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12073.894901] LNet: Removed LNI 192.168.204.150@tcp [12074.997735] Key type .llcrypt unregistered [12075.000965] Key type ._llcrypt unregistered [12097.065834] Key type ._llcrypt registered [12097.068827] Key type .llcrypt registered [12097.263873] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [12115.317665] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12117.176828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12117.238955] alg: No test for adler32 (adler32-zlib) [12118.512247] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [12118.926511] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [12120.719204] Key type lgssc registered [12122.634328] Lustre: Echo OBD driver; http://www.lustre.org/ [12165.900714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12165.921026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12167.377708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12167.427770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12167.535981] Lustre: lustre-MDT0000: new disk, initializing [12167.614135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12167.646175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12172.867887] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12184.632212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12184.769930] Lustre: 376163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12184.846661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12184.850252] Lustre: Skipped 1 previous similar message [12184.923266] Lustre: lustre-MDT0001: new disk, initializing [12185.056612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12185.149548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12185.163222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12190.117991] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12201.113598] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12208.187609] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12216.759387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12217.370062] Lustre: lustre-OST0000: new disk, initializing [12217.385330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12217.401531] Lustre: 378132:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12217.556494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12223.673279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12223.679848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12223.946495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12226.020442] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12236.946666] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12243.431250] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12243.754216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12250.064325] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12250.475235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12254.689966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12254.698328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12254.702112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12254.718183] Lustre: Skipped 1 previous similar message [12254.726123] Lustre: Skipped 1 previous similar message [12259.189233] Lustre: server umount lustre-OST0000 complete [12264.931096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12264.937954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12264.956632] Lustre: Skipped 1 previous similar message [12270.050531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12270.063441] Lustre: Skipped 2 previous similar messages [12270.461499] Lustre: server umount lustre-MDT0000 complete [12274.935450] LustreError: 376153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427225 with bad export cookie 15002232724930100912 [12274.950297] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12274.956500] LustreError: 376153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12275.354140] Lustre: server umount lustre-MDT0001 complete [12285.756937] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12288.089919] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:47:16 (1786427236) [12295.849622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12296.211563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12301.284546] LustreError: 380596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.322947] LustreError: 380596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12301.899707] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12306.403699] LustreError: 380597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12307.427825] LustreError: 380596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12312.191988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.777174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12319.459357] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12330.141728] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12338.084115] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12346.093032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12346.340813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12352.745641] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12360.762787] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12368.863602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12369.024467] Lustre: lustre-OST0001: new disk, initializing [12369.028636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12369.035150] Lustre: 383434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12369.147076] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12376.754790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12376.792541] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12377.083218] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12378.239344] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12389.931491] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12396.714551] Lustre: lustre-OST0001: Not available for connect from 192.168.204.50@tcp (stopping) [12397.537523] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12397.538394] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12397.553023] LustreError: Skipped 2 previous similar messages [12397.566989] Lustre: Skipped 1 previous similar message [12399.726473] Lustre: server umount lustre-OST0001 complete [12401.796891] LustreError: 382421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12407.791792] LustreError: 382421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12407.832423] LustreError: 382421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12418.021956] LustreError: 383740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12418.025864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12418.026438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12418.030283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12418.030293] Lustre: Skipped 2 previous similar messages [12418.046375] LustreError: 383740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12423.709584] Lustre: server umount lustre-OST0000 complete [12428.260925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12428.279260] Lustre: Skipped 1 previous similar message [12428.288312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12428.297910] Lustre: Skipped 4 previous similar messages [12434.058507] Lustre: server umount lustre-MDT0000 complete [12438.500283] LustreError: 380593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12438.549685] LustreError: 380593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12439.715396] LustreError: 380577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427390 with bad export cookie 15002232724930101794 [12439.720408] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12439.726902] LustreError: 380577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12454.367760] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12454.789629] Lustre: server umount lustre-MDT0001 complete [12470.612351] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12474.716889] Key type lgssc unregistered [12475.000277] LNet: 385429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12475.017711] LNetError: 385429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12475.049814] LNet: Removed LNI 192.168.204.150@tcp [12476.241408] Key type .llcrypt unregistered [12476.243478] Key type ._llcrypt unregistered [12546.802789] Key type ._llcrypt registered [12546.810793] Key type .llcrypt registered [12547.056612] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12549.646352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12549.785111] alg: No test for adler32 (adler32-zlib) [12551.183308] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [12551.542886] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [12553.231200] Key type lgssc registered [12555.140728] Lustre: Echo OBD driver; http://www.lustre.org/ [12564.886332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12564.900875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12566.206438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12566.246172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12566.377668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12574.090562] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12584.403392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12584.487585] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12584.517307] Lustre: 388231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12584.795224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12590.913189] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12599.241224] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12606.463204] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12610.530857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12610.539696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12610.553171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12611.049391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12611.067744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12611.078846] Lustre: Skipped 1 previous similar message [12614.838332] Lustre: server umount lustre-MDT0000 complete [12618.920983] LustreError: 388225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427569 with bad export cookie 4370967589789330384 [12618.928415] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12618.939588] LustreError: 388225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12619.261165] Lustre: server umount lustre-MDT0001 complete [12630.964895] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:52:59 (1786427579) [12638.326849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12638.799786] LustreError: 390744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.814108] LustreError: 390744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12638.864846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12643.842368] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12644.322724] LustreError: 390745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12649.445872] LustreError: 390744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.988033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12653.392625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12658.930564] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12668.125187] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12674.927987] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12682.160574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12682.474733] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12682.716978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12690.962529] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12699.900709] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12708.339852] Lustre: Permanently deactivating lustre-OST0000 [12708.373084] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12715.045153] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12716.579739] LustreError: 393685:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12716.589305] LustreError: 393685:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12716.601044] LustreError: 393685:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12719.879350] LustreError: 393734:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12719.891280] LustreError: 393734:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12724.702817] Lustre: server umount lustre-OST0000 complete [12728.172534] LustreError: 393983:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12728.182939] LustreError: 393983:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12731.877257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12731.904372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12731.915630] Lustre: Skipped 1 previous similar message [12736.992718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12737.000157] Lustre: Skipped 1 previous similar message [12741.196455] Lustre: server umount lustre-MDT0000 complete [12742.112924] LustreError: 390739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12742.147581] LustreError: 390739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12745.312640] LustreError: 390724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427695 with bad export cookie 4370967589789330965 [12745.324541] LustreError: 390724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12745.326925] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12755.067219] Lustre: server umount lustre-MDT0001 complete [12765.115065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12770.736973] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12773.695414] LustreError: 395144:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12780.593949] Lustre: 395483:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.150@tcp [12782.682442] Lustre: server umount MGS complete [12797.014710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12797.653487] LustreError: 396207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.683133] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12797.686816] Lustre: Skipped 1 previous similar message [12797.731448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12803.983953] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12808.164295] LustreError: 396208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12808.223362] LustreError: 396208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12815.306900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12815.715272] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12815.871887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12822.944901] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12833.283813] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12840.838660] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12843.589835] Lustre: Permanently reactivating lustre-OST0000 [12843.668853] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12843.685262] Lustre: Skipped 2 previous similar messages [12871.137560] LustreError: 397960:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12878.891885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12879.198498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12887.294668] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12896.180267] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12906.992323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12906.996329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12907.002493] LustreError: Skipped 1 previous similar message [12907.016883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12907.033031] Lustre: Skipped 2 previous similar messages [12907.062279] Lustre: Skipped 1 previous similar message [12909.446686] Lustre: server umount lustre-OST0000 complete [12917.218879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.221994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12917.233166] Lustre: Skipped 1 previous similar message [12917.265645] Lustre: Skipped 1 previous similar message [12920.564893] Lustre: server umount lustre-MDT0000 complete [12922.338469] LustreError: 396208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.362299] LustreError: 396208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12925.045389] LustreError: 396188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427875 with bad export cookie 4370967589789332015 [12925.056517] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12925.057418] LustreError: 396188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12925.534063] Lustre: server umount lustre-MDT0001 complete [12937.579069] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12940.877808] Key type lgssc unregistered [12941.518291] LNet: 400530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12941.529971] LNetError: 400530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12941.552371] LNet: Removed LNI 192.168.204.150@tcp [12942.513149] Key type .llcrypt unregistered [12942.517396] Key type ._llcrypt unregistered [12964.698829] Key type ._llcrypt registered [12964.701438] Key type .llcrypt registered [12964.816873] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [12987.923981] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12989.884701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12990.015336] alg: No test for adler32 (adler32-zlib) [12991.608678] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [12991.932668] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [12993.607212] Key type lgssc registered [12995.079279] Lustre: Echo OBD driver; http://www.lustre.org/ [13031.718815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13031.736491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13032.923665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13032.955973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13033.023453] Lustre: lustre-MDT0000: new disk, initializing [13033.132247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13033.177757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13037.688867] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13049.081448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13049.209424] Lustre: 403689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13049.286896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13049.292468] Lustre: Skipped 1 previous similar message [13049.366673] Lustre: lustre-MDT0001: new disk, initializing [13049.436631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13049.473514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13049.483873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13054.108279] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13063.516995] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13069.043772] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13075.925843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13076.209716] Lustre: lustre-OST0000: new disk, initializing [13076.214516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13076.228468] Lustre: 405659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13076.337406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13082.023539] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13084.252282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13084.271456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13084.475669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13091.497166] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13096.448831] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13096.970301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13101.860475] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13102.108304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13105.125879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13105.130668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13105.132463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13105.142708] Lustre: Skipped 1 previous similar message [13105.163882] LustreError: Skipped 1 previous similar message [13110.240434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13110.265907] Lustre: Skipped 2 previous similar messages [13110.728444] Lustre: server umount lustre-OST0000 complete [13115.363325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13115.377259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13115.381747] Lustre: Skipped 1 previous similar message [13120.484078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13120.494620] Lustre: Skipped 2 previous similar messages [13121.504286] Lustre: server umount lustre-MDT0000 complete [13125.800161] LustreError: 403680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786428076 with bad export cookie 2149530999769598185 [13125.810527] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13125.814334] LustreError: 403680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13126.176486] Lustre: server umount lustre-MDT0001 complete [13136.647290] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13138.623403] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:01:27 (1786428087) [13145.500789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13145.932612] LustreError: 408133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13145.967793] LustreError: 408133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13146.055825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13150.673496] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13151.201607] LustreError: 408133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13151.228429] LustreError: 408133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13156.322859] LustreError: 408134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13159.826023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13160.186344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13164.264927] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13172.228499] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13178.703550] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13185.382410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13185.600787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13191.887842] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13198.891281] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13200.637565] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13203.211136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13215.207500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13215.209525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13215.217606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13215.236475] Lustre: Skipped 1 previous similar message [13215.249748] Lustre: Skipped 1 previous similar message [13218.530766] Lustre: server umount lustre-OST0000 complete [13225.442064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13225.444860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13225.459374] Lustre: Skipped 1 previous similar message [13228.418349] Lustre: server umount lustre-MDT0000 complete [13230.566252] LustreError: 408133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13230.592223] LustreError: 408133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13232.030305] LustreError: 408114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786428182 with bad export cookie 2149530999769599067 [13232.034692] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13232.050611] LustreError: 408114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13232.362171] Lustre: server umount lustre-MDT0001 complete [13243.192973] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13245.909920] Key type lgssc unregistered [13246.237967] LNet: 411862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13246.244600] LNetError: 411862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13246.256693] LNet: Removed LNI 192.168.204.150@tcp [13247.173678] Key type .llcrypt unregistered [13247.179352] Key type ._llcrypt unregistered [13274.075984] Key type ._llcrypt registered [13274.079738] Key type .llcrypt registered [13274.216636] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13274.640530] Key type .llcrypt unregistered [13274.645343] Key type ._llcrypt unregistered [13293.381393] Key type ._llcrypt registered [13293.385383] Key type .llcrypt registered [13293.558482] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13294.453145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13294.520283] alg: No test for adler32 (adler32-zlib) [13295.791538] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [13296.115619] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13297.799169] Key type lgssc registered [13299.608509] Lustre: Echo OBD driver; http://www.lustre.org/ [13307.777664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13307.812260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13309.219166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13314.582935] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13318.636736] LustreError: 414083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13318.668597] LustreError: 414083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13323.576640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13323.748176] LustreError: 414082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13323.952763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13329.381372] LustreError: 414083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13329.405810] LustreError: 414083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13331.788838] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13333.481358] LustreError: 414082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13333.506880] LustreError: 414082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13333.512476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13338.609126] LustreError: 414083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13338.637861] LustreError: 414083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13340.738312] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13348.843580] LustreError: 414082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13348.895152] LustreError: 414082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13349.575169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13350.120243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13355.797293] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13365.222644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13365.234915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13365.259708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13365.749240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13365.785589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13366.839799] Lustre: server umount lustre-OST0000 complete [13371.483082] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13371.491191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13371.509965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13373.744153] Lustre: server umount lustre-MDT0000 complete [13375.970241] LustreError: 414082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13375.970634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13377.531352] LustreError: 414064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786428328 with bad export cookie 2178813517050655104 [13377.548577] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13377.566869] LustreError: 414064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13377.937770] Lustre: server umount lustre-MDT0001 complete [13388.131249] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13390.938247] Key type lgssc unregistered [13391.254461] LNet: 417233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13391.261856] LNetError: 417233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13392.294894] LNet: Removed LNI 192.168.204.150@tcp [13393.397631] Key type .llcrypt unregistered [13393.400815] Key type ._llcrypt unregistered [13422.054514] Key type ._llcrypt registered [13422.056067] Key type .llcrypt registered [13422.116938] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13423.055849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13423.153042] alg: No test for adler32 (adler32-zlib) [13424.130677] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [13424.422924] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13426.111299] Key type lgssc registered [13427.811121] Lustre: Echo OBD driver; http://www.lustre.org/ [13437.762969] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13437.776941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13453.279561] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13458.399401] LustreError: 418627:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.150@tcp: failed processing log, type 4: rc = -110 [13485.087730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13492.919637] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13502.421354] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13510.122356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13510.661378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13510.802413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13515.652233] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13515.753438] LustreError: 418652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13515.776064] LustreError: 418652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13519.852334] LustreError: 419663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13519.898284] LustreError: 419663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13524.969746] LustreError: 419665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13525.019167] LustreError: 419665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13526.706293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13527.098117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13534.167524] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13543.411549] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13550.968866] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13560.102510] Lustre: server umount lustre-OST0000 complete [13564.237048] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13564.255181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13564.292702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13567.970729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13567.972766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13573.092886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13573.099107] Lustre: Skipped 2 previous similar messages [13574.217961] Lustre: server umount lustre-MDT0000 complete [13578.220250] LustreError: 419668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13578.251315] LustreError: 419668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13578.606527] LustreError: 418635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786428529 with bad export cookie 5751658795758153734 [13578.623158] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13589.116219] Lustre: server umount lustre-MDT0001 complete [13601.191648] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13604.973835] Key type lgssc unregistered [13605.543868] LNet: 422205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13605.556906] LNetError: 422205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13605.577809] LNet: Removed LNI 192.168.204.150@tcp [13606.472429] Key type .llcrypt unregistered [13606.474188] Key type ._llcrypt unregistered [13634.419332] Key type ._llcrypt registered [13634.420916] Key type .llcrypt registered [13634.585554] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13637.041030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13637.127704] alg: No test for adler32 (adler32-zlib) [13638.196865] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [13638.503420] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13640.215400] Key type lgssc registered [13641.399789] Lustre: Echo OBD driver; http://www.lustre.org/ [13651.931708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13651.962471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13653.445591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13659.303266] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13662.691399] LustreError: 423624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13662.726622] LustreError: 423624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13667.815613] LustreError: 423625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13671.132574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13671.489922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13677.518347] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13686.740629] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13693.661103] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13702.460720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13702.791394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13710.148479] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13718.236932] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13723.256091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13728.740892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13728.742799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13728.755431] Lustre: Skipped 2 previous similar messages [13728.759737] Lustre: Skipped 1 previous similar message [13730.442571] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.50@tcp (stopping) [13730.446563] Lustre: Skipped 1 previous similar message [13733.870398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13733.878209] Lustre: Skipped 2 previous similar messages [13734.443540] Lustre: server umount lustre-MDT0000 complete [13735.553186] LustreError: 423621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13736.806173] LustreError: 423604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.50@tcp arrived at 1786428687 with bad export cookie 2062766655474163936 [13736.826774] LustreError: 423604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13738.977421] LustreError: 423625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13739.011838] LustreError: 423625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13739.147186] LustreError: 423606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786428689 with bad export cookie 2062766655474163271 [13739.185196] LustreError: 423606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13739.188752] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13742.050917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13742.061496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13742.082333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13744.107626] LustreError: 423620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13744.150243] LustreError: 423620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13745.449665] Lustre: server umount lustre-OST0000 complete [13753.631549] Lustre: server umount lustre-MDT0001 complete [13765.276322] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13769.099756] Key type lgssc unregistered [13769.650200] LNet: 427314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13769.658284] LNetError: 427314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13769.681693] LNet: Removed LNI 192.168.204.150@tcp [13770.796632] Key type .llcrypt unregistered [13770.806335] Key type ._llcrypt unregistered [13800.393143] Key type ._llcrypt registered [13800.400744] Key type .llcrypt registered [13800.529423] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13801.550320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13801.757762] alg: No test for adler32 (adler32-zlib) [13802.893530] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [13803.136582] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13804.807199] Key type lgssc registered [13806.275213] Lustre: Echo OBD driver; http://www.lustre.org/ [13816.236615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13816.255612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13817.751662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13823.294563] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13828.068060] LustreError: 428733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13828.115763] LustreError: 428733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13831.658640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13831.926969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13836.822821] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13844.032536] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13850.507493] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13859.565326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13860.019358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13863.146804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13868.535503] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13877.628428] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13885.898788] Lustre: server umount lustre-MDT0001 complete [13886.187851] LustreError: 428728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13887.971244] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13887.985965] LustreError: 430916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13887.991523] Lustre: Skipped 1 previous similar message [13888.015300] LustreError: 430916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13893.103197] LustreError: 428733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13894.113611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13894.123551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13894.148992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13898.211176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13898.211819] LustreError: 428733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13898.235116] LustreError: 428733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13899.653288] Lustre: server umount lustre-OST0000 complete [13905.191819] Lustre: server umount lustre-MDT0000 complete [13924.437748] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13928.321436] Key type lgssc unregistered [13928.871848] LNet: 432423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13928.883452] LNetError: 432423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13928.906664] LNet: Removed LNI 192.168.204.150@tcp [13929.897218] Key type .llcrypt unregistered [13929.899996] Key type ._llcrypt unregistered [13963.649033] Key type ._llcrypt registered [13963.655331] Key type .llcrypt registered [13963.812487] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [13982.844458] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13984.956886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13985.035163] alg: No test for adler32 (adler32-zlib) [13986.092993] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [13986.369744] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13988.063181] Key type lgssc registered [13989.809260] Lustre: Echo OBD driver; http://www.lustre.org/ [14038.030285] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [14057.644680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14057.673227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14058.878238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14058.896879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14058.958187] Lustre: lustre-MDT0000: new disk, initializing [14059.044535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14059.065085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14064.621857] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14081.600739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14081.881264] Lustre: lustre-OST0000: new disk, initializing [14081.884631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14081.888866] Lustre: Skipped 1 previous similar message [14081.892380] Lustre: 437462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14081.989661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14088.924597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14088.938223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14088.961818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14090.919273] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14103.042437] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14122.608783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14122.742300] Lustre: 436443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14122.808609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14122.896380] Lustre: lustre-MDT0001: new disk, initializing [14123.030531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14123.072444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14123.086607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14128.926600] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14141.258973] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14141.503876] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14146.365362] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14146.596651] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14156.956645] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.50@tcp (stopping) [14157.287611] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14157.296213] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14157.307457] Lustre: Skipped 1 previous similar message [14158.886228] Lustre: server umount lustre-MDT0001 complete [14167.523278] LustreError: 436456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14167.560873] LustreError: 436456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14172.644853] LustreError: 436452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14172.661787] LustreError: 436452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14177.780577] LustreError: 436450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14177.853834] LustreError: 436450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14181.346861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14181.358124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14181.384431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14181.395366] Lustre: Skipped 2 previous similar messages [14182.883353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14182.889071] LustreError: 436455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14185.917739] Lustre: server umount lustre-OST0000 complete [14188.015300] LustreError: 436456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14197.347583] Lustre: server umount lustre-MDT0000 complete [14214.062945] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14216.853357] Key type lgssc unregistered [14217.195365] LNet: 441217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14217.204806] LNetError: 441217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14217.227614] LNet: Removed LNI 192.168.204.150@tcp [14218.439745] Key type .llcrypt unregistered [14218.446065] Key type ._llcrypt unregistered [14269.301604] Key type ._llcrypt registered [14269.303749] Key type .llcrypt registered [14269.430102] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14269.794297] Key type .llcrypt unregistered [14269.803101] Key type ._llcrypt unregistered [14290.760432] Key type ._llcrypt registered [14290.763548] Key type .llcrypt registered [14290.874845] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [14291.858106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14292.578036] alg: No test for adler32 (adler32-zlib) [14293.614047] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [14293.839748] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [14295.655823] Key type lgssc registered [14297.549971] Lustre: Echo OBD driver; http://www.lustre.org/ [14328.484831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14328.517796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14329.928610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14329.967984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14330.045238] Lustre: lustre-MDT0000: new disk, initializing [14330.105377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14330.120197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14336.521904] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14348.987545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14349.193732] Lustre: 444602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14349.250241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14349.255181] Lustre: Skipped 1 previous similar message [14349.394812] Lustre: lustre-MDT0001: new disk, initializing [14349.544510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14349.575551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14349.585798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14355.577592] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14367.840532] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14375.230726] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14384.291569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14384.615425] Lustre: lustre-OST0000: new disk, initializing [14384.619381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14384.628095] Lustre: 446572:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14384.769883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14393.788210] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14394.100119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14394.127856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14394.483961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14406.414542] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14414.821690] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14414.826753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14414.845370] Lustre: Skipped 2 previous similar messages [14414.864566] Lustre: Skipped 2 previous similar messages [14419.946484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14419.968791] Lustre: Skipped 2 previous similar messages [14420.097483] Lustre: server umount lustre-MDT0000 complete [14424.475958] LustreError: 444593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786429375 with bad export cookie 17039006748603345509 [14424.485509] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14424.498864] LustreError: 444593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14424.774327] Lustre: server umount lustre-MDT0001 complete [14440.087819] Lustre: server umount lustre-OST0000 complete [14452.356599] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:23:20 (1786429400) [14461.876256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14462.392986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14467.554573] LustreError: 448604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14467.579297] LustreError: 448604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14468.742881] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14471.666555] LustreError: 448605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14476.772042] LustreError: 448604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14479.270983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14479.867241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14485.771712] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14495.347468] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14502.728146] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14524.716189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14524.910792] LustreError: 450430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14524.993624] Lustre: Found index 0 for lustre-OST0000, updating log [14525.190023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14531.637315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14533.358081] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14542.223730] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14549.997667] Lustre: server umount lustre-OST0000 complete [14567.328030] Lustre: 443134:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786429501/real 1786429501] req@000000008f478a84 x1873206831033088/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786429517 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14567.364022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14568.735804] Lustre: 443133:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786429502/real 1786429502] req@0000000098cb0f17 x1873206831034112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786429518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14572.001595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14572.007646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14572.024875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14572.831942] Lustre: 443132:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786429507/real 1786429507] req@0000000024bc75a0 x1873206831035392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786429523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14576.613496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14576.639300] Lustre: Skipped 3 previous similar messages [14577.631433] Lustre: 443133:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786429512/real 1786429512] req@00000000abfb0031 x1873206831036544/t0(0) o400->lustre-OST0000-osc-MDT0001@1.2.3.4@tcp:28/4 lens 224/224 e 0 to 1 dl 1786429528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14582.239214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14582.440735] Lustre: server umount lustre-MDT0000 complete [14582.764920] LustreError: 449333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14582.822931] LustreError: 449333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14587.255654] LustreError: 448586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786429537 with bad export cookie 17039006748603346895 [14587.270276] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14587.284597] LustreError: 448586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14587.699134] Lustre: server umount lustre-MDT0001 complete [14594.975321] LNet: 1 local NIs in recovery (showing 1): 192.168.204.150@tcp [14599.332345] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:25:47 (1786429547) [15233.101942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15233.854517] LustreError: 455021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15234.012285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15239.137738] LustreError: 455022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15240.872697] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15244.261904] LustreError: 455021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15250.412368] LustreError: 455021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15250.439769] LustreError: 455021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15251.713803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15251.974256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15257.048731] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15266.111750] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15272.756713] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15286.176309] LNet: 1 local NIs in recovery (showing 1): 192.168.204.150@tcp [15294.450731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15294.804829] Lustre: Found index 0 for lustre-OST0000, updating log [15295.410800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15304.267854] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15312.651516] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15314.914399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15314.927045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15314.942342] Lustre: Skipped 1 previous similar message [15314.949768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15314.956452] Lustre: Skipped 2 previous similar messages [15329.247138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15329.502956] Lustre: server umount lustre-OST0000 complete [15335.008723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15335.016751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15335.045310] Lustre: Skipped 1 previous similar message [15335.064903] Lustre: Skipped 1 previous similar message [15336.101146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15336.105791] Lustre: Skipped 1 previous similar message [15338.154623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15338.167510] Lustre: Skipped 3 previous similar messages [15340.201683] LustreError: 455022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15340.276852] Lustre: server umount lustre-MDT0000 complete [15344.296278] LustreError: 455003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786430294 with bad export cookie 17039006748603347987 [15344.305912] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15344.317646] LustreError: 455003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15344.762555] Lustre: server umount lustre-MDT0001 complete [15345.567264] LNet: 1 local NIs in recovery (showing 1): 192.168.204.150@tcp [15379.276536] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [15392.468075] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [15440.119778] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:39:47 (1786430387) [15464.753788] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [15468.756315] Key type lgssc unregistered [15469.073983] LNet: 461644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15469.082428] LNetError: 461644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15558.184829] LNet: Removed LNI 192.168.204.150@tcp [15559.331637] Key type .llcrypt unregistered [15559.333902] Key type ._llcrypt unregistered [15586.416134] Key type ._llcrypt registered [15586.421705] Key type .llcrypt registered [15586.651174] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [15589.158296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15590.025272] alg: No test for adler32 (adler32-zlib) [15591.640642] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [15591.925490] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [15593.687205] Key type lgssc registered [15595.973288] Lustre: Echo OBD driver; http://www.lustre.org/ [15603.099843] LNet: Added LNI 192.168.204.150@tcp100 [8/256/0/180] [15604.427235] LNet: Added LNI 192.168.204.150@tcp101 [8/256/0/180] [15605.401817] LNet: Added LNI 192.168.204.150@tcp102 [8/256/0/180] [15606.545103] LNet: Added LNI 192.168.204.150@tcp103 [8/256/0/180] [15607.594534] LNet: Added LNI 192.168.204.150@tcp104 [8/256/0/180] [15608.730577] LNet: Added LNI 192.168.204.150@tcp105 [8/256/0/180] [15609.738842] LNet: Added LNI 192.168.204.150@tcp106 [8/256/0/180] [15610.413778] LNet: Added LNI 192.168.204.150@tcp107 [8/256/0/180] [15611.488906] LNet: Added LNI 192.168.204.150@tcp108 [8/256/0/180] [15612.852574] LNet: Added LNI 192.168.204.150@tcp109 [8/256/0/180] [15613.657571] LNet: Added LNI 192.168.204.150@tcp110 [8/256/0/180] [15615.029769] LNet: Added LNI 192.168.204.150@tcp111 [8/256/0/180] [15616.374264] LNet: Added LNI 192.168.204.150@tcp112 [8/256/0/180] [15617.626739] LNet: Added LNI 192.168.204.150@tcp113 [8/256/0/180] [15618.769484] LNet: Added LNI 192.168.204.150@tcp114 [8/256/0/180] [15620.082875] LNet: Added LNI 192.168.204.150@tcp115 [8/256/0/180] [15621.089204] LNet: Added LNI 192.168.204.150@tcp116 [8/256/0/180] [15622.056644] LNet: Added LNI 192.168.204.150@tcp117 [8/256/0/180] [15623.274727] LNet: Added LNI 192.168.204.150@tcp118 [8/256/0/180] [15624.879895] LNet: Added LNI 192.168.204.150@tcp119 [8/256/0/180] [15626.174479] LNet: Added LNI 192.168.204.150@tcp120 [8/256/0/180] [15627.160413] LNet: Added LNI 192.168.204.150@tcp121 [8/256/0/180] [15628.310980] LNet: Added LNI 192.168.204.150@tcp122 [8/256/0/180] [15629.877971] LNet: Added LNI 192.168.204.150@tcp123 [8/256/0/180] [15631.030181] LNet: Added LNI 192.168.204.150@tcp124 [8/256/0/180] [15632.145902] LNet: Added LNI 192.168.204.150@tcp125 [8/256/0/180] [15633.531109] LNet: Added LNI 192.168.204.150@tcp126 [8/256/0/180] [15634.919596] LNet: Added LNI 192.168.204.150@tcp127 [8/256/0/180] [15636.271586] LNet: Added LNI 192.168.204.150@tcp128 [8/256/0/180] [15637.476569] LNet: Added LNI 192.168.204.150@tcp129 [8/256/0/180] [15638.979157] LNet: Added LNI 192.168.204.150@tcp130 [8/256/0/180] [15640.018540] LNet: Added LNI 192.168.204.150@tcp131 [8/256/0/180] [15641.387265] LNet: Added LNI 192.168.204.150@tcp132 [8/256/0/180] [15642.854412] LNet: Added LNI 192.168.204.150@tcp133 [8/256/0/180] [15644.110701] LNet: Added LNI 192.168.204.150@tcp134 [8/256/0/180] [15645.257741] LNet: Added LNI 192.168.204.150@tcp135 [8/256/0/180] [15654.930140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15654.980975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15656.127333] Lustre: 464682:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.150@tcp: can't add NID 192.168.204.150@tcp130: rc = -75 [15656.246253] LNetError: 462138:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.150 [15656.282295] LNetError: Protocol error connecting to 192.168.204.150@tcp113 at host 192.168.204.150:988: is it running a compatible version of Lustre? [15656.324450] LNetError: Skipped 8 previous similar messages [15656.479389] LNet: 6 local NIs in recovery (showing 6): 192.168.204.150@tcp124, 192.168.204.150@tcp126, 192.168.204.150@tcp127, 192.168.204.150@tcp129, 192.168.204.150@tcp125, 192.168.204.150@tcp128 [15656.562051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15656.604596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15656.680909] Lustre: lustre-MDT0000: new disk, initializing [15656.775928] Lustre: 464739:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.150@tcp: can't add NID 192.168.204.150@tcp130: rc = -75 [15656.787047] Lustre: 464739:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15656.808131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15656.849592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15657.339693] LNetError: 462137:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.150 [15657.374877] LNetError: 462137:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [15657.385929] LNetError: Protocol error connecting to 192.168.204.150@tcp114 at host 192.168.204.150:988: is it running a compatible version of Lustre? [15657.422421] LNetError: Skipped 3 previous similar messages [15659.618676] LNetError: 464711:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.150 [15659.665081] LNetError: 464711:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [15659.691920] LNetError: Protocol error connecting to 192.168.204.150@tcp124 at host 192.168.204.150:988: is it running a compatible version of Lustre? [15659.710944] LNetError: Skipped 8 previous similar messages [15663.714464] LNetError: 462136:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.150 [15663.748238] LNetError: 462136:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15663.772622] LNetError: Protocol error connecting to 192.168.204.150@tcp124 at host 192.168.204.150:988: is it running a compatible version of Lustre? [15664.657308] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15671.908480] LNetError: 464703:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.150 [15671.932813] LNetError: Protocol error connecting to 192.168.204.150@tcp124 at host 192.168.204.150:988: is it running a compatible version of Lustre? [15678.985065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15679.183285] Lustre: 464698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15679.243324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15679.250093] Lustre: Skipped 1 previous similar message [15679.341783] Lustre: lustre-MDT0001: new disk, initializing [15679.396776] Lustre: 465503:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.150@tcp: can't add NID 192.168.204.150@tcp130: rc = -75 [15679.409253] Lustre: 465503:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15679.658431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15679.774195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15679.808435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15685.514962] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15687.147172] LNetError: 464716:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.150 [15687.168630] LNetError: Protocol error connecting to 192.168.204.150@tcp113 at host 192.168.204.150:988: is it running a compatible version of Lustre? [15687.251668] Lustre: 465837:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.150@tcp: can't add NID 192.168.204.150@tcp130: rc = -75 [15687.266544] Lustre: 465837:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15698.781977] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15705.569228] LNetError: 462137:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.150 [15705.587792] LNetError: 462137:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [15705.598167] LNetError: Protocol error connecting to 192.168.204.150@tcp114 at host 192.168.204.150:988: is it running a compatible version of Lustre? [15705.603772] LNetError: Skipped 12 previous similar messages [15706.269811] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15716.633657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15716.961223] LNet: 1 local NIs in recovery (showing 1): 192.168.204.150@tcp124 [15717.241560] Lustre: lustre-OST0000: new disk, initializing [15717.250270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15717.273785] Lustre: 466677:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15717.399476] Lustre: 466682:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.150@tcp: can't add NID 192.168.204.150@tcp130: rc = -75 [15717.405431] Lustre: 466682:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15717.475173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15722.690485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15722.732598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15723.086257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15727.392187] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15740.464997] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15743.467368] LNetError: 462138:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.150 [15743.478974] LNetError: 462138:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 23 previous similar messages [15743.494844] LNetError: Protocol error connecting to 192.168.204.150@tcp114 at host 192.168.204.150:988: is it running a compatible version of Lustre? [15743.513056] LNetError: Skipped 23 previous similar messages [15753.705730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15753.706383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15753.717903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15753.732168] Lustre: Skipped 1 previous similar message [15753.742411] LustreError: Skipped 1 previous similar message [15757.248799] Lustre: server umount lustre-OST0000 complete [15763.940293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15763.947723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15763.968529] Lustre: Skipped 1 previous similar message [15763.990653] Lustre: Skipped 2 previous similar messages [15769.073960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15769.084786] Lustre: Skipped 1 previous similar message [15774.183243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15774.187020] Lustre: Skipped 1 previous similar message [15777.247545] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15777.555874] Lustre: server umount lustre-MDT0000 complete [15782.494041] LustreError: 464691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786430733 with bad export cookie 4170749512097872489 [15782.504020] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15782.954781] Lustre: server umount lustre-MDT0001 complete [15797.479249] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [15800.849150] Key type lgssc unregistered [15801.258249] LNet: 468598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15801.273144] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15801.285824] LNet: Removed LNI 192.168.204.150@tcp [15802.336755] LNet: Removed LNI 192.168.204.150@tcp100 [15802.341525] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15802.350737] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15802.367686] LNet: Removed LNI 192.168.204.150@tcp101 [15803.423401] LNet: Removed LNI 192.168.204.150@tcp102 [15803.429594] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15803.442284] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15803.453448] LNet: Removed LNI 192.168.204.150@tcp103 [15804.511988] LNet: Removed LNI 192.168.204.150@tcp104 [15805.537585] LNet: Removed LNI 192.168.204.150@tcp105 [15805.545703] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15805.556770] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15805.579416] LNet: Removed LNI 192.168.204.150@tcp106 [15806.623738] LNet: Removed LNI 192.168.204.150@tcp107 [15807.647204] LNet: Removed LNI 192.168.204.150@tcp108 [15808.672996] LNet: Removed LNI 192.168.204.150@tcp109 [15809.695173] LNet: Removed LNI 192.168.204.150@tcp110 [15809.704206] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15809.716769] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15809.732275] LNet: Removed LNI 192.168.204.150@tcp111 [15810.783186] LNet: Removed LNI 192.168.204.150@tcp112 [15811.807959] LNet: Removed LNI 192.168.204.150@tcp113 [15812.831199] LNet: Removed LNI 192.168.204.150@tcp114 [15813.855235] LNet: Removed LNI 192.168.204.150@tcp115 [15814.880678] LNet: Removed LNI 192.168.204.150@tcp116 [15815.904337] LNet: Removed LNI 192.168.204.150@tcp117 [15816.927175] LNet: Removed LNI 192.168.204.150@tcp118 [15817.951185] LNet: Removed LNI 192.168.204.150@tcp119 [15817.957864] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15817.972190] LNetError: 468598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15817.991163] LNet: Removed LNI 192.168.204.150@tcp120 [15819.039194] LNet: Removed LNI 192.168.204.150@tcp121 [15820.063174] LNet: Removed LNI 192.168.204.150@tcp122 [15821.087598] LNet: Removed LNI 192.168.204.150@tcp123 [15821.093477] LNet: Removed LNI 192.168.204.150@tcp124 [15822.111356] LNet: Removed LNI 192.168.204.150@tcp125 [15823.135272] LNet: Removed LNI 192.168.204.150@tcp126 [15824.159180] LNet: Removed LNI 192.168.204.150@tcp127 [15825.183225] LNet: Removed LNI 192.168.204.150@tcp128 [15826.207229] LNet: Removed LNI 192.168.204.150@tcp129 [15826.211534] LNet: Removed LNI 192.168.204.150@tcp130 [15826.214781] LNet: Removed LNI 192.168.204.150@tcp131 [15826.218239] LNet: Removed LNI 192.168.204.150@tcp132 [15826.220464] LNet: Removed LNI 192.168.204.150@tcp133 [15826.223198] LNet: Removed LNI 192.168.204.150@tcp134 [15826.233791] LNet: Removed LNI 192.168.204.150@tcp135 [15826.999165] Key type .llcrypt unregistered [15827.000624] Key type ._llcrypt unregistered [15855.584594] Key type ._llcrypt registered [15855.586869] Key type .llcrypt registered [15855.761966] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [15874.513289] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [15876.558101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15876.837576] alg: No test for adler32 (adler32-zlib) [15877.925814] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [15878.197804] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [15879.855375] Key type lgssc registered [15882.189465] Lustre: Echo OBD driver; http://www.lustre.org/ [15925.453314] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:47:53 (1786430873) [15947.463478] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [15950.694476] Key type lgssc unregistered [15951.124227] LNet: 472694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15951.135565] LNetError: 472694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15951.149062] LNet: Removed LNI 192.168.204.150@tcp [15952.253163] Key type .llcrypt unregistered [15952.255912] Key type ._llcrypt unregistered [15973.090863] Key type ._llcrypt registered [15973.094620] Key type .llcrypt registered [15973.226764] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [15975.071507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15975.118635] alg: No test for adler32 (adler32-zlib) [15976.351874] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [15976.651767] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [15978.559314] Key type lgssc registered [15980.497659] Lustre: Echo OBD driver; http://www.lustre.org/ [15992.327429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15992.346649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15993.675725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15993.710746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15993.772878] Lustre: lustre-MDT0000: new disk, initializing [15993.849045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15993.879245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15999.989165] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16012.815381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16012.946606] Lustre: 473909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16013.007299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16013.014793] Lustre: Skipped 1 previous similar message [16013.080632] Lustre: lustre-MDT0001: new disk, initializing [16013.164275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16013.201528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16013.226352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16018.350266] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16030.164385] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16037.103819] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16044.584618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16044.830570] Lustre: lustre-OST0000: new disk, initializing [16044.834635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16044.841177] Lustre: 475880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16044.930815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16052.527908] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16053.333856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16053.354471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16053.519300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16062.814312] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16067.803960] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing lnet_if_list [16073.160308] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16073.204899] LNet: Added LNI 192.168.204.150@tcp10 [8/256/0/180] [16079.575937] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16079.598923] LNet: Added LNI 192.168.204.150@tcp11 [8/256/0/180] [16084.052915] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16084.077971] LNet: Added LNI 192.168.204.150@tcp12 [8/256/0/180] [16089.453742] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16089.493621] LNet: Added LNI 192.168.204.150@tcp13 [8/256/0/180] [16094.836619] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16094.879875] LNet: Added LNI 192.168.204.150@tcp14 [8/256/0/180] [16099.741792] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16099.778781] LNet: Added LNI 192.168.204.150@tcp15 [8/256/0/180] [16106.727559] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16106.764782] LNet: Added LNI 192.168.204.150@tcp16 [8/256/0/180] [16112.578848] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16112.607664] LNet: Added LNI 192.168.204.150@tcp17 [8/256/0/180] [16117.846610] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16117.878629] LNet: Added LNI 192.168.204.150@tcp18 [8/256/0/180] [16124.004411] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16124.027903] LNet: Added LNI 192.168.204.150@tcp19 [8/256/0/180] [16130.185045] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16130.254572] LNet: Added LNI 192.168.204.150@tcp20 [8/256/0/180] [16136.406609] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16136.445230] LNet: Added LNI 192.168.204.150@tcp21 [8/256/0/180] [16143.038750] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16143.067331] LNet: Added LNI 192.168.204.150@tcp22 [8/256/0/180] [16150.281719] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16150.313865] LNet: Added LNI 192.168.204.150@tcp23 [8/256/0/180] [16158.279508] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16158.335238] LNet: Added LNI 192.168.204.150@tcp24 [8/256/0/180] [16165.461520] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16165.527890] LNet: Added LNI 192.168.204.150@tcp25 [8/256/0/180] [16174.146370] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16174.192868] LNet: Added LNI 192.168.204.150@tcp26 [8/256/0/180] [16181.831988] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16181.898070] LNet: Added LNI 192.168.204.150@tcp27 [8/256/0/180] [16188.810481] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16188.843633] LNet: Added LNI 192.168.204.150@tcp28 [8/256/0/180] [16195.498992] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16195.534899] LNet: Added LNI 192.168.204.150@tcp29 [8/256/0/180] [16204.009267] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16204.060029] LNet: Added LNI 192.168.204.150@tcp30 [8/256/0/180] [16211.992867] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16212.028185] LNet: Added LNI 192.168.204.150@tcp31 [8/256/0/180] [16219.907393] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16219.958270] LNet: Added LNI 192.168.204.150@tcp32 [8/256/0/180] [16227.640772] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16227.665553] LNet: Added LNI 192.168.204.150@tcp33 [8/256/0/180] [16236.643744] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16236.708260] LNet: Added LNI 192.168.204.150@tcp34 [8/256/0/180] [16243.325505] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16243.359428] LNet: Added LNI 192.168.204.150@tcp35 [8/256/0/180] [16251.740941] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16251.778635] LNet: Added LNI 192.168.204.150@tcp36 [8/256/0/180] [16259.326853] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16259.371419] LNet: Added LNI 192.168.204.150@tcp37 [8/256/0/180] [16266.737553] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16266.761350] LNet: Added LNI 192.168.204.150@tcp38 [8/256/0/180] [16273.267807] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16273.289344] LNet: Added LNI 192.168.204.150@tcp39 [8/256/0/180] [16281.926162] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16281.963279] LNet: Added LNI 192.168.204.150@tcp40 [8/256/0/180] [16289.580519] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16289.630484] LNet: Added LNI 192.168.204.150@tcp41 [8/256/0/180] [16296.918979] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16296.954510] LNet: Added LNI 192.168.204.150@tcp42 [8/256/0/180] [16305.016808] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16305.054366] LNet: Added LNI 192.168.204.150@tcp43 [8/256/0/180] [16313.238835] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16313.272616] LNet: Added LNI 192.168.204.150@tcp44 [8/256/0/180] [16321.880139] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16322.016074] LNet: Added LNI 192.168.204.150@tcp45 [8/256/0/180] [16331.080421] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16331.182212] LNet: Added LNI 192.168.204.150@tcp46 [8/256/0/180] [16341.438711] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16341.464707] LNet: Added LNI 192.168.204.150@tcp47 [8/256/0/180] [16351.456887] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16351.489356] LNet: Added LNI 192.168.204.150@tcp48 [8/256/0/180] [16360.801420] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16360.836859] LNet: Added LNI 192.168.204.150@tcp49 [8/256/0/180] [16373.854691] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16373.904259] LNet: Added LNI 192.168.204.150@tcp50 [8/256/0/180] [16396.772079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16396.776022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16396.787116] Lustre: Skipped 2 previous similar messages [16396.796542] Lustre: Skipped 1 previous similar message [16401.628809] Lustre: server umount lustre-MDT0000 complete [16406.801815] LustreError: 473902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786431357 with bad export cookie 7398324315901471287 [16406.817437] LustreError: 473902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16406.818270] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16407.021501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16407.022273] LustreError: 476148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16407.058215] LustreError: 476148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16407.244600] Lustre: server umount lustre-MDT0001 complete [16428.930901] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [16437.837940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16452.004825] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x66ac20c46e71342b [16452.015092] Lustre: MGC192.168.204.150@tcp: Connection restored to 0@lo (at 0@lo) [16452.339226] LustreError: 475872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16452.388487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16457.448990] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16457.707400] LustreError: 475873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16460.780504] LustreError: 475872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16460.792837] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16465.895201] LustreError: 475873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16467.357450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16467.669182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16470.775482] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16472.798458] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16482.454468] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16490.114029] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16517.097580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16517.115320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16517.153070] Lustre: Skipped 1 previous similar message [16517.155581] Lustre: Skipped 1 previous similar message [16522.220390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16522.222900] Lustre: Skipped 2 previous similar messages [16527.333351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16530.400698] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16530.632107] Lustre: server umount lustre-OST0000 complete [16537.569713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16537.592402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16537.601125] Lustre: Skipped 2 previous similar messages [16547.812495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16547.830492] Lustre: Skipped 3 previous similar messages [16549.343394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16549.510938] Lustre: server umount lustre-MDT0000 complete [16552.930112] LustreError: 501734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16552.964519] LustreError: 501734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16554.218171] LustreError: 474715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786431504 with bad export cookie 7398324315901473835 [16554.233880] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16554.667888] Lustre: server umount lustre-MDT0001 complete [16568.521815] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [16571.962269] Key type lgssc unregistered [16572.499333] LNet: 502938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16572.513489] LNetError: 502938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16572.521858] LNet: Removed LNI 192.168.204.150@tcp [16572.526718] LNet: Removed LNI 192.168.204.150@tcp10 [16572.529764] LNet: Removed LNI 192.168.204.150@tcp11 [16572.534476] LNet: Removed LNI 192.168.204.150@tcp12 [16572.537335] LNet: Removed LNI 192.168.204.150@tcp13 [16572.540425] LNet: Removed LNI 192.168.204.150@tcp14 [16572.543020] LNet: Removed LNI 192.168.204.150@tcp15 [16572.546402] LNet: Removed LNI 192.168.204.150@tcp16 [16572.549610] LNet: Removed LNI 192.168.204.150@tcp17 [16572.552433] LNet: Removed LNI 192.168.204.150@tcp18 [16572.554935] LNet: Removed LNI 192.168.204.150@tcp19 [16572.557613] LNet: Removed LNI 192.168.204.150@tcp20 [16572.562602] LNet: Removed LNI 192.168.204.150@tcp21 [16572.565566] LNet: Removed LNI 192.168.204.150@tcp22 [16572.568756] LNet: Removed LNI 192.168.204.150@tcp23 [16572.572328] LNet: Removed LNI 192.168.204.150@tcp24 [16572.576308] LNet: Removed LNI 192.168.204.150@tcp25 [16572.578291] LNet: Removed LNI 192.168.204.150@tcp26 [16572.582606] LNet: Removed LNI 192.168.204.150@tcp27 [16572.585619] LNet: Removed LNI 192.168.204.150@tcp28 [16572.588922] LNet: Removed LNI 192.168.204.150@tcp29 [16572.591615] LNet: Removed LNI 192.168.204.150@tcp30 [16572.594650] LNet: Removed LNI 192.168.204.150@tcp31 [16572.597565] LNet: Removed LNI 192.168.204.150@tcp32 [16572.600641] LNet: Removed LNI 192.168.204.150@tcp33 [16572.603250] LNet: Removed LNI 192.168.204.150@tcp34 [16572.606251] LNet: Removed LNI 192.168.204.150@tcp35 [16572.608543] LNet: Removed LNI 192.168.204.150@tcp36 [16572.611440] LNet: Removed LNI 192.168.204.150@tcp37 [16572.614448] LNet: Removed LNI 192.168.204.150@tcp38 [16572.617334] LNet: Removed LNI 192.168.204.150@tcp39 [16572.620361] LNet: Removed LNI 192.168.204.150@tcp40 [16572.622769] LNet: Removed LNI 192.168.204.150@tcp41 [16572.626360] LNet: Removed LNI 192.168.204.150@tcp42 [16572.629177] LNet: Removed LNI 192.168.204.150@tcp43 [16572.635437] LNet: Removed LNI 192.168.204.150@tcp44 [16572.639450] LNet: Removed LNI 192.168.204.150@tcp45 [16572.641684] LNet: Removed LNI 192.168.204.150@tcp46 [16572.644589] LNet: Removed LNI 192.168.204.150@tcp47 [16572.649564] LNet: Removed LNI 192.168.204.150@tcp48 [16572.653406] LNet: Removed LNI 192.168.204.150@tcp49 [16572.666290] LNet: Removed LNI 192.168.204.150@tcp50 [16573.429239] Key type .llcrypt unregistered [16573.431390] Key type ._llcrypt unregistered [16597.284791] Key type ._llcrypt registered [16597.287203] Key type .llcrypt registered [16597.455059] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [16620.081697] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [16621.939230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16622.442080] alg: No test for adler32 (adler32-zlib) [16623.927607] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [16624.201441] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [16626.000730] Key type lgssc registered [16627.702949] Lustre: Echo OBD driver; http://www.lustre.org/ [16672.825661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16672.870671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16674.131517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16674.153236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16674.217163] Lustre: lustre-MDT0000: new disk, initializing [16674.299733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16674.316066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16678.527161] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16689.678825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16689.888587] Lustre: 506143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16689.961162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16689.973257] Lustre: Skipped 1 previous similar message [16690.052752] Lustre: lustre-MDT0001: new disk, initializing [16690.165263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16690.210049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16690.234145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16695.142622] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16705.427377] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16712.520146] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16722.492422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16722.775391] Lustre: *** cfs_fail_loc=90f, val=0*** [16722.869395] Lustre: lustre-OST0000: new disk, initializing [16722.872778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16722.879763] Lustre: 508175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16723.014265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16730.030407] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16732.757111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16732.771094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16732.903656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16741.320658] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16753.635175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16753.643494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16753.660203] LustreError: Skipped 1 previous similar message [16753.663473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16753.695862] Lustre: Skipped 1 previous similar message [16755.832290] Lustre: server umount lustre-OST0000 complete [16761.829028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16761.839731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16761.867286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16761.874802] Lustre: Skipped 1 previous similar message [16763.874602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16763.885093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16763.918727] Lustre: Skipped 1 previous similar message [16766.539991] Lustre: server umount lustre-MDT0000 complete [16770.885769] LustreError: 506133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786431721 with bad export cookie 1449608165647309217 [16770.899666] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16770.910541] LustreError: 506133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16771.248593] Lustre: server umount lustre-MDT0001 complete [16783.026213] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [16785.600919] Key type lgssc unregistered [16785.910692] LNet: 510085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16785.916099] LNetError: 510085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16785.929184] LNet: Removed LNI 192.168.204.150@tcp [16786.950435] Key type .llcrypt unregistered [16786.952393] Key type ._llcrypt unregistered [16806.593433] Key type ._llcrypt registered [16806.594742] Key type .llcrypt registered [16806.700133] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [16808.028307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16808.052163] alg: No test for adler32 (adler32-zlib) [16809.198713] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [16809.524302] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [16811.175215] Key type lgssc registered [16812.269130] Lustre: Echo OBD driver; http://www.lustre.org/ [16820.246649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16820.269957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16821.715517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16826.028830] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16831.970945] LustreError: 511360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16831.988455] LustreError: 511360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16835.359027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16835.703273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16840.744434] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16848.528849] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16854.492677] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16861.478993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16861.784413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16869.358529] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16877.264542] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16891.226153] loop0: detected capacity change from 0 to 204800000 [16891.606556] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16891.622111] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16891.632833] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16897.915147] loop0: detected capacity change from 0 to 204800000 [16898.254241] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16898.267509] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16898.277596] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16903.613607] loop0: detected capacity change from 0 to 204800000 [16909.341247] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16909.585404] Lustre: fs15246-OST0000: new disk, initializing [16909.591811] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16909.602168] Lustre: 514954:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16909.661683] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16916.260123] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16926.115695] loop1: detected capacity change from 0 to 204800000 [16932.897351] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16933.021416] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16933.065191] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16933.150723] Lustre: fs15246-MDT0000: new disk, initializing [16933.383250] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16933.417680] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16933.535253] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16934.570603] Lustre: 516076:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000077a212c0 x1873209469221504/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786431901 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16938.701290] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [16939.696739] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16939.713938] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16939.778493] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16965.096791] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16965.129860] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16970.216160] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16971.081964] Lustre: server umount fs15246-MDT0000 complete [16975.341539] LustreError: 512364:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16980.455566] LustreError: 511356:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16982.405368] Lustre: server umount fs15246-OST0000 complete [16999.394694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16999.401395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16999.414728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17000.417217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17000.422629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17002.576484] Lustre: server umount lustre-OST0000 complete [17009.633339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17009.642725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17009.661576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17009.669469] Lustre: Skipped 2 previous similar messages [17016.289306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17016.293722] Lustre: Skipped 3 previous similar messages [17021.410616] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17021.554537] Lustre: server umount lustre-MDT0000 complete [17025.515024] LustreError: 511359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17025.539751] LustreError: 511359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17025.695635] LustreError: 511339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786431976 with bad export cookie 7489917655417118426 [17025.703261] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17025.707801] LustreError: 511339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17026.047593] Lustre: server umount lustre-MDT0001 complete [17039.090777] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [17043.013717] Key type lgssc unregistered [17043.266918] LNet: 519178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17043.275796] LNetError: 519178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17043.304572] LNet: Removed LNI 192.168.204.150@tcp [17044.588903] Key type .llcrypt unregistered [17044.592113] Key type ._llcrypt unregistered [17065.541615] Key type ._llcrypt registered [17065.544029] Key type .llcrypt registered [17065.668874] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [17067.055462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17067.138835] alg: No test for adler32 (adler32-zlib) [17068.222413] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [17068.446978] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [17070.096608] Key type lgssc registered [17071.615901] Lustre: Echo OBD driver; http://www.lustre.org/ [17116.593954] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [17127.544694] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [17165.150617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17165.186306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17166.396955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17166.436281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17166.564379] Lustre: lustre-MDT0000: new disk, initializing [17166.661416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17166.695264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17171.425534] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [17182.622899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17182.914077] Lustre: 523648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17182.996371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17183.000820] Lustre: Skipped 1 previous similar message [17183.074874] Lustre: lustre-MDT0001: new disk, initializing [17183.196152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17183.256309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17183.268322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17188.396477] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [17198.893635] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17205.791738] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17213.138765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17213.348786] Lustre: lustre-OST0000: new disk, initializing [17213.352085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17213.357772] Lustre: 525619:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17213.450507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17219.878619] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [17220.674097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17220.705543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17220.888217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17229.595300] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17235.939474] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17236.213651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17241.555370] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17241.847929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17246.700579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17246.712263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17246.713265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17246.723842] Lustre: Skipped 1 previous similar message [17250.610316] Lustre: server umount lustre-OST0000 complete [17256.928323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17256.954906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17256.957798] Lustre: Skipped 1 previous similar message [17261.811644] Lustre: server umount lustre-MDT0000 complete [17266.476821] LustreError: 523641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786432217 with bad export cookie 12146779412131857011 [17266.488333] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17266.496250] LustreError: 523641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17266.848513] Lustre: server umount lustre-MDT0001 complete [17297.940363] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [17313.671517] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [17352.735395] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16916 sec ========= 03:11:41 (1786432301) [17355.264414] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:11:43 (1786432303) === [17360.098378] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:11:48 (1786432308) === [17406.910511] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [17410.715084] Key type lgssc unregistered [17411.123910] LNet: 532291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17411.138858] LNetError: 532291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17411.159942] LNet: Removed LNI 192.168.204.150@tcp [17412.311837] Key type .llcrypt unregistered [17412.317144] Key type ._llcrypt unregistered