[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 503511923 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002248] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007593] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011125] LSM: Security Framework initializing [ 0.013046] Yama: becoming mindful. [ 0.014045] SELinux: Initializing. [ 0.015077] *** VALIDATE selinux *** [ 0.023589] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027395] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029103] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] *** VALIDATE tmpfs *** [ 0.032421] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.041948] debug: unmapping init [mem 0xffffffffb4459000-0xffffffffb4460fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044725] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051019] ... event mask: 000000070000000f [ 0.052322] rcu: Hierarchical SRCU implementation. [ 0.054785] smp: Bringing up secondary CPUs ... [ 0.055815] x86: Booting SMP configuration: [ 0.056030] .... node #0, CPUs: #1 #2 #3 [ 0.059546] smp: Brought up 1 node, 4 CPUs [ 0.061016] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.126732] node 0 deferred pages initialised in 62ms [ 0.130106] devtmpfs: initialized [ 0.131275] x86/mm: Memory block size: 128MB [ 0.133971] gcov: version magic: 0x41383552 [ 0.135233] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136067] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137282] pinctrl core: initialized pinctrl subsystem [ 0.138201] [ 0.138823] ************************************************************* [ 0.139016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140013] ** ** [ 0.141014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142014] ** ** [ 0.143011] ** This means that this kernel is built to expose internal ** [ 0.144015] ** IOMMU data structures, which may compromise security on ** [ 0.145012] ** your system. ** [ 0.146015] ** ** [ 0.147012] ** If you see this message and you are not debugging the ** [ 0.148012] ** kernel, report this immediately to your vendor! ** [ 0.149015] ** ** [ 0.150013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151012] ************************************************************* [ 0.152818] NET: Registered protocol family 16 [ 0.153461] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.154068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.155065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.157010] cpuidle: using governor menu [ 0.158788] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.161587] PCI: Using configuration type 1 for base access [ 0.163178] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.174020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.175346] cryptd: max_cpu_qlen set to 1000 [ 0.177272] ACPI: Added _OSI(Module Device) [ 0.178013] ACPI: Added _OSI(Processor Device) [ 0.179013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.180016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.184058] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.186581] ACPI: Interpreter enabled [ 0.187087] ACPI: PM: (supports S0 S3 S4 S5) [ 0.188013] ACPI: Using IOAPIC for interrupt routing [ 0.189116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.190353] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.198466] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.199058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.200073] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.201077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.203331] acpiphp: Slot [2] registered [ 0.204163] acpiphp: Slot [5] registered [ 0.205154] acpiphp: Slot [6] registered [ 0.206088] acpiphp: Slot [7] registered [ 0.207141] acpiphp: Slot [8] registered [ 0.208115] acpiphp: Slot [9] registered [ 0.209116] acpiphp: Slot [10] registered [ 0.210134] acpiphp: Slot [3] registered [ 0.211137] acpiphp: Slot [4] registered [ 0.212089] acpiphp: Slot [11] registered [ 0.213091] acpiphp: Slot [12] registered [ 0.214099] acpiphp: Slot [13] registered [ 0.215120] acpiphp: Slot [14] registered [ 0.216096] acpiphp: Slot [15] registered [ 0.217116] acpiphp: Slot [16] registered [ 0.218099] acpiphp: Slot [17] registered [ 0.219092] acpiphp: Slot [18] registered [ 0.220087] acpiphp: Slot [19] registered [ 0.221100] acpiphp: Slot [20] registered [ 0.222081] acpiphp: Slot [21] registered [ 0.223114] acpiphp: Slot [22] registered [ 0.224104] acpiphp: Slot [23] registered [ 0.225116] acpiphp: Slot [24] registered [ 0.226103] acpiphp: Slot [25] registered [ 0.227098] acpiphp: Slot [26] registered [ 0.228104] acpiphp: Slot [27] registered [ 0.229099] acpiphp: Slot [28] registered [ 0.230142] acpiphp: Slot [29] registered [ 0.231121] acpiphp: Slot [30] registered [ 0.232090] acpiphp: Slot [31] registered [ 0.233057] PCI host bridge to bus 0000:00 [ 0.234019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.235022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.236029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.237028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.238035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.243046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.245494] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.251106] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.256260] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.272019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.277059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.281022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.284018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.287069] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.291584] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.297317] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.301171] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.306828] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.315779] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.337017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.346014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.352650] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.364020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.372019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.436042] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.490012] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.498024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.508017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.533017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.551227] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.568016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.589088] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.608762] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.619021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.634021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.659039] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.673283] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.685018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.694027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.715030] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.731931] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.734487] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.737478] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.740342] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.742279] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.748073] iommu: Default domain type: Passthrough [ 0.749000] SCSI subsystem initialized [ 0.751205] ACPI: bus type USB registered [ 0.753628] usbcore: registered new interface driver usbfs [ 0.756244] usbcore: registered new interface driver hub [ 0.759126] usbcore: registered new device driver usb [ 0.761255] pps_core: LinuxPPS API ver. 1 registered [ 0.764011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.768066] PTP clock support registered [ 0.770432] EDAC MC: Ver: 3.0.0 [ 0.773174] PCI: Using ACPI for IRQ routing [ 0.775418] NetLabel: Initializing [ 0.776000] NetLabel: domain hash size = 128 [ 0.780029] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.782129] NetLabel: unlabeled traffic allowed by default [ 0.786159] vgaarb: loaded [ 0.788196] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.791021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.798687] clocksource: Switched to clocksource kvm-clock [ 0.982715] VFS: Disk quotas dquot_6.6.0 [ 0.987834] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.993119] *** VALIDATE ramfs *** [ 0.995712] *** VALIDATE hugetlbfs *** [ 0.998213] pnp: PnP ACPI init [ 1.003195] pnp: PnP ACPI: found 6 devices [ 1.030615] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.033814] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.037470] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.041349] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.044564] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.047507] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.051421] NET: Registered protocol family 2 [ 1.059824] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.068329] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.073565] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.086234] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.093051] TCP: Hash tables configured (established 65536 bind 65536) [ 1.096351] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.102019] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.107693] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.111987] NET: Registered protocol family 1 [ 1.115441] RPC: Registered named UNIX socket transport module. [ 1.117664] RPC: Registered udp transport module. [ 1.119813] RPC: Registered tcp transport module. [ 1.121798] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.124348] NET: Registered protocol family 44 [ 1.126162] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.128062] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.130084] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.132548] PCI: CLS 0 bytes, default 64 [ 1.134303] Unpacking initramfs... [ 2.823561] debug: unmapping init [mem 0xffff98813cc54000-0xffff98813ffbffff] [ 2.832598] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.836829] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.841443] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.448207] Initialise system trusted keyrings [ 3.450042] Key type blacklist registered [ 3.454628] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.469442] zbud: loaded [ 3.473054] *** VALIDATE nfs *** [ 3.474415] *** VALIDATE nfs4 *** [ 3.476687] pstore: using deflate compression [ 3.481677] Platform Keyring initialized [ 3.633544] NET: Registered protocol family 38 [ 3.635454] Key type asymmetric registered [ 3.637389] Asymmetric key parser 'x509' registered [ 3.639662] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.642877] io scheduler mq-deadline registered [ 3.645629] io scheduler kyber registered [ 3.648186] io scheduler bfq registered [ 3.650431] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.654584] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.657808] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.660763] ACPI: Power Button [PWRF] [ 3.667350] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.674965] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.691099] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.699439] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.733824] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.767680] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.800774] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.805970] Non-volatile memory driver v1.3 [ 3.807512] Linux agpgart interface v0.103 [ 3.845965] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.850257] vda: detected capacity change from 0 to 74649600 [ 3.872526] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.875622] vdb: detected capacity change from 0 to 1073741824 [ 3.904796] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.907905] vdc: detected capacity change from 0 to 2621440000 [ 3.948346] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.951774] vdd: detected capacity change from 0 to 2621440000 [ 3.982689] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.987820] vde: detected capacity change from 0 to 4294967296 [ 4.008970] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.012024] vdf: detected capacity change from 0 to 4294967296 [ 4.020375] libphy: Fixed MDIO Bus: probed [ 4.028773] usbcore: registered new interface driver usbserial_generic [ 4.031927] usbserial: USB Serial support registered for generic [ 4.034763] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.040598] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.042815] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.045375] mousedev: PS/2 mouse device common for all mice [ 4.049753] rtc_cmos 00:05: RTC can wake from S4 [ 4.052734] rtc_cmos 00:05: registered as rtc0 [ 4.057260] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.059985] intel_pstate: CPU model not supported [ 4.062168] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.065613] hid: raw HID events driver (C) Jiri Kosina [ 4.067399] usbcore: registered new interface driver usbhid [ 4.069732] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.070681] usbhid: USB HID core driver [ 4.070820] drop_monitor: Initializing network drop monitor service [ 4.070962] Initializing XFRM netlink socket [ 4.071332] NET: Registered protocol family 10 [ 4.075504] Segment Routing with IPv6 [ 4.079901] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.080461] NET: Registered protocol family 17 [ 4.083641] mpls_gso: MPLS GSO support [ 4.093721] RAS: Correctable Errors collector initialized. [ 4.095683] AVX version of gcm_enc/dec engaged. [ 4.097204] AES CTR mode by8 optimization enabled [ 4.180236] sched_clock: Marking stable (4180201134, 0)->(5209549140, -1029348006) [ 4.184596] registered taskstats version 1 [ 4.187628] Loading compiled-in X.509 certificates [ 4.190565] zswap: loaded using pool lzo/zbud [ 4.214178] Key type big_key registered [ 4.229112] Key type encrypted registered [ 4.231557] ima: No TPM chip found, activating TPM-bypass! [ 4.233480] ima: Allocated hash algorithm: sha1 [ 4.235177] ima: No architecture policies found [ 4.236617] evm: Initialising EVM extended attributes: [ 4.238275] evm: security.selinux [ 4.239470] evm: security.ima [ 4.240436] evm: security.capability [ 4.241499] evm: HMAC attrs: 0x1 [ 4.243807] rtc_cmos 00:05: setting system clock to 2026-07-26 14:38:14 UTC (1785076694) [ 4.251184] debug: unmapping init [mem 0xffffffffb5403000-0xffffffffb55fffff] [ 4.253761] debug: unmapping init [mem 0xffffffffb4182000-0xffffffffb4458fff] [ 4.264083] Write protecting the kernel read-only data: 28672k [ 4.267102] debug: unmapping init [mem 0xffffffffb2803000-0xffffffffb29fffff] [ 4.269338] debug: unmapping init [mem 0xffffffffb3114000-0xffffffffb31fffff] [ 4.301989] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.309422] systemd[1]: Detected virtualization kvm. [ 4.311458] systemd[1]: Detected architecture x86-64. [ 4.313161] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.345661] systemd[1]: No hostname configured. [ 4.348778] systemd[1]: Set hostname to . [ 4.350816] random: systemd: uninitialized urandom read (16 bytes read) [ 4.355364] systemd[1]: Initializing machine ID from random generator. [ 4.436980] random: ln: uninitialized urandom read (6 bytes read) [ 4.576052] random: systemd: uninitialized urandom read (16 bytes read) [ 4.578792] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.583569] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.589875] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.472980] device-mapper: uevent: version 1.0.3 [ 5.475397] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager...[ 5.919852] random: fast init done [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.854992] virtio_net virtio0 ens2: renamed from eth0 [ 7.365883] scsi host0: ata_piix [ 7.434540] scsi host1: ata_piix [ 7.439784] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.445900] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.757916] random: crng init done [ 11.763095] random: 7 urandom warning(s) missed due to ratelimiting [ 14.495455] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.403147] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.377790] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.785374] SELinux: Disabled at runtime. [ 17.869844] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.881753] systemd[1]: Detected virtualization kvm. [ 17.887880] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.742422] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.749350] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.764783] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.770815] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.775789] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.791226] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.799806] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 18.871674] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.780179] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.454452] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.662793] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.033220] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.126287] EDAC sbridge: Ver: 1.1.2 [ 24.625660] Key type dns_resolver registered [ 25.009340] NFS: Registering the id_resolver key type [ 25.011332] Key type id_resolver registered [ 25.012874] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg401-server login: [ 81.927968] libcfs: loading out-of-tree module taints kernel. [ 82.010754] Key type ._llcrypt registered [ 82.015421] Key type .llcrypt registered [ 82.198961] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 105.204423] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 107.014605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.041042] alg: No test for adler32 (adler32-zlib) [ 108.462270] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 109.315348] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 111.144623] Key type lgssc registered [ 112.657427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.993481] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 150.112345] hrtimer: interrupt took 7224408 ns [ 181.203244] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 197.052635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 197.097365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.520543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 198.557309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.692320] Lustre: lustre-MDT0000: new disk, initializing [ 198.815558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.842591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 203.799567] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.405827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.580598] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.650174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 219.666773] Lustre: Skipped 1 previous similar message [ 219.788784] Lustre: lustre-MDT0001: new disk, initializing [ 219.915322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.957312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 219.971652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.469931] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.581311] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 242.909650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 243.311598] Lustre: lustre-OST0000: new disk, initializing [ 243.318518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 243.329730] Lustre: 8459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 243.448760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.720892] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.424468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.435739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.529786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 268.124543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.310117] Lustre: lustre-OST0001: new disk, initializing [ 268.318688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 268.323460] Lustre: 9531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.419823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.423674] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.551583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 276.567959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 276.637084] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 290.477861] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 302.623586] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 310.131483] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing check_logdir /tmp/testlogs/ [ 315.896577] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing yml_node [ 321.464606] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 324.114071] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 327.439199] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 329.375428] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jul 26 10:43:38 EDT 2026 [ 349.034744] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 350.998980] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 368.613035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 368.624208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.672841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.242837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.258364] Lustre: Skipped 7 previous similar messages [ 374.668448] Lustre: server umount lustre-MDT0000 complete [ 379.145365] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077069 with bad export cookie 12774943075089413823 [ 379.146460] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 379.168964] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 379.367494] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.388638] Lustre: Skipped 3 previous similar messages [ 379.392419] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 384.481472] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.484168] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 384.512773] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 384.520300] Lustre: Skipped 2 previous similar messages [ 386.012051] Lustre: server umount lustre-MDT0001 complete [ 391.274367] Lustre: server umount lustre-OST0000 complete [ 395.905587] Lustre: server umount lustre-OST0001 complete [ 403.884832] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 456.826100] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 502.411839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.720657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 502.775486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 502.843922] Lustre: lustre-MDT0000: new disk, initializing [ 502.894777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.909588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 507.211721] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 518.546314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.689650] Lustre: 15110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 518.782246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 518.787001] Lustre: Skipped 1 previous similar message [ 518.949945] Lustre: lustre-MDT0001: new disk, initializing [ 519.067906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 519.133598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 519.148579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 524.423121] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 535.478815] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 542.201381] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 549.496658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 549.829501] Lustre: lustre-OST0000: new disk, initializing [ 549.833472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 549.847863] Lustre: 17080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 549.977195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.691142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 551.697876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 551.872855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 557.071104] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 567.714101] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 573.548607] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 573.784184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 578.439647] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 578.860657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 582.624255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 582.637988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.659560] Lustre: Skipped 1 previous similar message [ 582.666695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 582.671030] Lustre: Skipped 1 previous similar message [ 587.051360] Lustre: server umount lustre-OST0000 complete [ 590.821945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 590.831483] LustreError: Skipped 1 previous similar message [ 590.833750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.842844] Lustre: Skipped 1 previous similar message [ 590.846055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 605.151176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 605.428960] Lustre: server umount lustre-MDT0000 complete [ 608.225551] LustreError: 15912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.242237] LustreError: 15912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 609.134441] LustreError: 15102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077299 with bad export cookie 12774943075089415356 [ 609.147199] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.157411] LustreError: 15102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 609.505728] Lustre: server umount lustre-MDT0001 complete [ 614.691984] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:48:23 (1785077303) [ 617.684147] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 619.739983] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:48:29 (1785077309) [ 627.739233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.281431] LustreError: 19492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.375550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.795473] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 633.825095] LustreError: 19493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.930743] LustreError: 19492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.800978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.100525] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 658.360789] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 664.961722] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 673.650541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 681.697318] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 691.399877] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 710.116938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 710.126868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.145751] Lustre: Skipped 1 previous similar message [ 710.151572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 710.155392] Lustre: Skipped 6 previous similar messages [ 716.309207] Lustre: server umount lustre-OST0000 complete [ 721.380331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 721.393323] LustreError: Skipped 1 previous similar message [ 721.403623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 721.422594] Lustre: Skipped 1 previous similar message [ 727.180639] Lustre: server umount lustre-MDT0000 complete [ 728.549887] LustreError: 20496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 728.573347] LustreError: 20496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 732.193557] LustreError: 19472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077422 with bad export cookie 12774943075089416238 [ 732.196854] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 732.227074] LustreError: 19472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 732.461788] Lustre: server umount lustre-MDT0001 complete [ 747.450668] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 751.233656] Key type lgssc unregistered [ 751.639043] LNet: 23047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.649921] LNetError: 23047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.665464] LNet: Removed LNI 192.168.204.101@tcp [ 752.491168] Key type .llcrypt unregistered [ 752.492804] Key type ._llcrypt unregistered [ 779.221846] Key type ._llcrypt registered [ 779.225872] Key type .llcrypt registered [ 779.326930] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 780.417375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 780.449123] alg: No test for adler32 (adler32-zlib) [ 781.847551] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 782.193313] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 783.919216] Key type lgssc registered [ 785.269868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 794.729158] loop: module loaded [ 794.825497] loop0: detected capacity change from 0 to 204800000 [ 795.031324] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 795.057952] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 795.318793] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 801.817945] loop0: detected capacity change from 0 to 204800000 [ 801.982850] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 801.998811] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 802.012603] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 808.001797] loop0: detected capacity change from 0 to 204800000 [ 813.893172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 813.915631] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.183332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 815.265047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 815.340311] Lustre: lustre-MDT0000: new disk, initializing [ 815.450536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.484991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 819.777620] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 829.400279] loop1: detected capacity change from 0 to 204800000 [ 835.619835] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.056823] Lustre: lustre-OST07c6: new disk, initializing [ 836.062189] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 836.065241] Lustre: Skipped 1 previous similar message [ 836.076050] Lustre: 26234:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 836.145441] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 841.790784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 841.796161] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 841.827198] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 843.262145] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 849.524906] Lustre: 25224:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 849.539857] Lustre: 25224:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 849.544732] Lustre: 25224:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 849.550890] Lustre: 25224:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 849.557543] Lustre: 25224:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 849.566986] Lustre: 25224:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 865.780343] Lustre: 25224:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 865.794412] Lustre: 25224:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 865.800893] Lustre: 25224:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 865.808665] Lustre: 25224:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 865.819695] Lustre: 25224:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 865.825766] Lustre: 25224:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 871.072784] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 871.091264] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 871.099520] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 871.104949] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 871.125981] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 871.135243] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 871.140118] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 871.146697] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 871.151311] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 871.157402] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 871.164800] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 871.171588] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.464332] Lustre: 25223:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 874.475419] Lustre: 25223:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 874.481207] Lustre: 25223:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 874.488037] Lustre: 25223:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.493199] Lustre: 25223:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 874.502604] Lustre: 25223:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.507496] Lustre: 25223:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 874.518605] Lustre: 25223:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.527967] Lustre: 25223:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 874.532362] Lustre: 25223:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 874.541705] Lustre: 25223:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 874.558855] Lustre: 25223:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.491098] Lustre: 25223:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 879.501169] Lustre: 25223:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 879.509607] Lustre: 25223:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 879.524242] Lustre: 25223:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.530041] Lustre: 25223:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 879.537654] Lustre: 25223:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.547521] Lustre: 25223:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 879.556575] Lustre: 25223:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.570642] Lustre: 25223:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 879.592729] Lustre: 25223:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 879.599716] Lustre: 25223:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 879.608649] Lustre: 25223:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 887.892302] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 887.909716] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 887.919308] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 887.925453] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 887.937905] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 887.947144] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 887.960242] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 887.971201] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 887.981751] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 887.988101] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 887.998635] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 888.006494] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 938.979070] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 938.984109] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.024098] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 944.096189] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 949.222020] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 953.313660] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 953.505986] Lustre: server umount lustre-OST07c6 complete [ 963.217869] Lustre: server umount lustre-MDT0000 complete [ 982.211755] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 985.290883] Key type lgssc unregistered [ 985.595797] LNet: 28797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.609882] LNetError: 28797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.632692] LNet: Removed LNI 192.168.204.101@tcp [ 986.575394] Key type .llcrypt unregistered [ 986.578413] Key type ._llcrypt unregistered [ 1007.344475] Key type ._llcrypt registered [ 1007.348823] Key type .llcrypt registered [ 1007.513945] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1008.686065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.815996] alg: No test for adler32 (adler32-zlib) [ 1010.019847] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1010.188030] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1011.831208] Key type lgssc registered [ 1013.464863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1024.869120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1024.889887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.953249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.213759] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1037.295516] LustreError: 30069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1037.320828] LustreError: 30069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1042.405510] LustreError: 30070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.522414] LustreError: 30069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.513200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1049.175083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1056.024832] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1066.097294] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1072.599114] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1079.996183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1080.298907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.898439] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1094.758387] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1097.719315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1118.178251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1118.189867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.195813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1118.205084] Lustre: Skipped 1 previous similar message [ 1118.218072] Lustre: Skipped 1 previous similar message [ 1122.007635] Lustre: server umount lustre-OST0000 complete [ 1128.420592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.447221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1128.458089] Lustre: Skipped 1 previous similar message [ 1133.440900] Lustre: server umount lustre-MDT0000 complete [ 1133.539424] LustreError: 32676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.573432] LustreError: 32676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1137.577885] LustreError: 32854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077827 with bad export cookie 17379218742230218932 [ 1137.584637] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1137.587337] LustreError: 32854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1138.081707] Lustre: server umount lustre-MDT0001 complete [ 1152.369948] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1156.028872] Key type lgssc unregistered [ 1156.434286] LNet: 33808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.457354] LNetError: 33808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.490919] LNet: Removed LNI 192.168.204.101@tcp [ 1157.615403] Key type .llcrypt unregistered [ 1157.617619] Key type ._llcrypt unregistered [ 1180.177656] Key type ._llcrypt registered [ 1180.188907] Key type .llcrypt registered [ 1180.364525] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1181.758336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1182.024740] alg: No test for adler32 (adler32-zlib) [ 1183.197932] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1183.498360] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1185.159327] Key type lgssc registered [ 1186.547771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1196.437197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1196.456567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.075619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.552214] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1208.290677] LustreError: 35083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.312558] LustreError: 35083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1213.409791] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.119383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.741782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1220.731423] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1231.044861] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1238.898738] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1248.601842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1249.329897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.883679] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1267.320670] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1268.795744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1284.071545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1284.076824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.081491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1284.099115] Lustre: Skipped 1 previous similar message [ 1284.117066] Lustre: Skipped 1 previous similar message [ 1288.686358] Lustre: server umount lustre-OST0000 complete [ 1294.307228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.312241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.332470] Lustre: Skipped 1 previous similar message [ 1294.352521] Lustre: Skipped 1 previous similar message [ 1299.424853] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.465777] Lustre: server umount lustre-MDT0000 complete [ 1304.552231] LustreError: 35083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.557644] LustreError: 35063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785077994 with bad export cookie 17723157235492536156 [ 1304.581532] LustreError: 35083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1304.617807] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.083937] Lustre: server umount lustre-MDT0001 complete [ 1319.411418] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1323.322828] Key type lgssc unregistered [ 1323.784849] LNet: 38645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.791548] LNetError: 38645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.810289] LNet: Removed LNI 192.168.204.101@tcp [ 1324.956181] Key type .llcrypt unregistered [ 1324.958696] Key type ._llcrypt unregistered [ 1353.575175] Key type ._llcrypt registered [ 1353.576961] Key type .llcrypt registered [ 1353.690669] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1355.006398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.056947] alg: No test for adler32 (adler32-zlib) [ 1356.272734] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1356.507362] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1358.239259] Key type lgssc registered [ 1360.022069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1370.288157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1370.310761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.132506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.390650] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1382.384134] LustreError: 40074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.407101] LustreError: 40074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1387.488459] LustreError: 40075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.476796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.897515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1394.168565] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1402.440588] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1408.817504] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1416.633837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1416.942879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.996275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1423.899538] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1432.509889] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1439.714458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.717270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.729016] Lustre: Skipped 2 previous similar messages [ 1439.736118] Lustre: Skipped 1 previous similar message [ 1440.299487] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.1@tcp (stopping) [ 1440.310347] Lustre: Skipped 1 previous similar message [ 1444.847298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.859749] Lustre: Skipped 2 previous similar messages [ 1445.766335] Lustre: server umount lustre-MDT0000 complete [ 1449.963367] LustreError: 40803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.989342] LustreError: 40803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1450.220082] LustreError: 40043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078140 with bad export cookie 5976045530621418800 [ 1450.223307] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1450.240989] LustreError: 40043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1450.517739] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.1@tcp (stopping) [ 1450.528688] Lustre: Skipped 1 previous similar message [ 1453.489185] LustreError: 40070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.519122] LustreError: 40070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1455.073839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.096059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1455.104172] Lustre: Skipped 1 previous similar message [ 1456.884289] Lustre: server umount lustre-MDT0001 complete [ 1472.413595] Lustre: server umount lustre-OST0000 complete [ 1493.547449] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1497.594422] Key type lgssc unregistered [ 1498.002838] LNet: 43934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.013525] LNetError: 43934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1498.040967] LNet: Removed LNI 192.168.204.101@tcp [ 1499.181512] Key type .llcrypt unregistered [ 1499.183460] Key type ._llcrypt unregistered [ 1525.663345] Key type ._llcrypt registered [ 1525.666858] Key type .llcrypt registered [ 1525.789875] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1527.052752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1527.102427] alg: No test for adler32 (adler32-zlib) [ 1528.522433] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1528.833973] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1530.575221] Key type lgssc registered [ 1532.309702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1542.415219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1542.443360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.975839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.728112] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1554.406792] LustreError: 45356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.420736] LustreError: 45356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1559.523867] LustreError: 45357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1561.725686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.178865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1568.167528] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1578.613803] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1586.610908] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1595.215734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1595.741429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1604.303233] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1614.263739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1614.558305] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1622.499696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1622.521591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1622.542956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1624.547259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1624.555597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1624.576388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.622554] Lustre: Skipped 2 previous similar messages [ 1627.630490] Lustre: lustre-OST0000: Not available for connect from 192.168.204.1@tcp (stopping) [ 1629.683490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1629.696280] Lustre: Skipped 1 previous similar message [ 1634.785811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1634.796768] Lustre: Skipped 2 previous similar messages [ 1636.319207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1636.493186] Lustre: server umount lustre-OST0000 complete [ 1654.959493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.984635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1659.371085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1659.382086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.564169] Lustre: server umount lustre-MDT0000 complete [ 1665.524137] LustreError: 45356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.544151] LustreError: 45356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1666.565153] LustreError: 48058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078356 with bad export cookie 10913663687532764290 [ 1666.575796] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1666.576901] LustreError: 48058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1673.182734] Lustre: server umount lustre-MDT0001 complete [ 1684.551680] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1687.656200] Key type lgssc unregistered [ 1688.009686] LNet: 49067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.024621] LNetError: 49067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.046491] LNet: Removed LNI 192.168.204.101@tcp [ 1689.150227] Key type .llcrypt unregistered [ 1689.152768] Key type ._llcrypt unregistered [ 1715.982161] Key type ._llcrypt registered [ 1715.992135] Key type .llcrypt registered [ 1716.221947] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1718.144228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1718.170593] alg: No test for adler32 (adler32-zlib) [ 1719.368267] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1719.716676] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1721.471588] Key type lgssc registered [ 1722.882115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1732.240904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1732.261609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.855297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.676395] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1744.364856] LustreError: 50484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.380709] LustreError: 50484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1748.822740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.488143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1755.029841] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1762.939083] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1769.405891] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1778.253736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1778.568402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.575567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1787.678060] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1798.852143] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1804.529145] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1808.101933] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1821.435192] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-2611h08m29s [ 1823.751436] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1829.861254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.865184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.909660] Lustre: Skipped 1 previous similar message [ 1829.932769] Lustre: Skipped 2 previous similar messages [ 1830.948428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.986275] Lustre: Skipped 2 previous similar messages [ 1832.553816] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.1@tcp (stopping) [ 1832.565371] Lustre: Skipped 1 previous similar message [ 1832.872691] Lustre: server umount lustre-MDT0000 complete [ 1837.029041] LustreError: 50484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.046077] LustreError: 50484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1838.620751] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1839.085512] LustreError: 50481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.183178] LustreError: 50481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.203575] LustreError: 50481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1846.453790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.709495] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1847.135991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.202581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1848.175628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1853.469064] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1857.489706] Lustre: DEBUG MARKER: Wait for df (36767) ... [ 1859.612297] Lustre: DEBUG MARKER: done [ 1865.699700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.712309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.730052] Lustre: Skipped 1 previous similar message [ 1865.750270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.745328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.755634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.085981] Lustre: server umount lustre-OST0000 complete [ 1879.010397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1879.015838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.017553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.017559] Lustre: Skipped 3 previous similar messages [ 1879.048639] Lustre: Skipped 1 previous similar message [ 1882.193196] Lustre: server umount lustre-MDT0000 complete [ 1886.182263] LustreError: 50481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.209608] LustreError: 50481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1887.971030] LustreError: 50465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078578 with bad export cookie 10306994718916061432 [ 1887.972665] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1887.983026] LustreError: 50465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1888.516848] Lustre: server umount lustre-MDT0001 complete [ 1901.161233] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1905.904788] Key type lgssc unregistered [ 1906.516072] LNet: 55411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.533473] LNetError: 55411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.554164] LNet: Removed LNI 192.168.204.101@tcp [ 1907.807950] Key type .llcrypt unregistered [ 1907.810943] Key type ._llcrypt unregistered [ 1956.440836] Key type ._llcrypt registered [ 1956.444246] Key type .llcrypt registered [ 1956.567825] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1958.011608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.028158] alg: No test for adler32 (adler32-zlib) [ 1959.222253] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1959.475984] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1961.200825] Key type lgssc registered [ 1963.198310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1973.751661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1973.789993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.972779] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1974.991902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1975.109861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.739800] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1988.866123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.956251] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1988.988512] Lustre: 57622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1989.353997] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1995.250895] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2004.536139] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2011.429522] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2021.110580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2021.536104] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2021.919655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.221670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2029.906172] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2038.635220] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2046.426426] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-2611h12m15s [ 2048.043168] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2055.619687] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2058.098329] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2058.186886] LustreError: 57628:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2078.754216] Lustre: lustre-MDT0000: Client 695e9c17-a38f-403b-8e51-5ac7d7a20d28 (at 192.168.204.1@tcp) reconnecting [ 2098.287149] LustreError: 57628:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2100.922675] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2103.625236] Lustre: DEBUG MARKER: done [ 2110.436118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2110.437215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.445735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2110.452216] LustreError: Skipped 1 previous similar message [ 2110.480240] Lustre: Skipped 1 previous similar message [ 2115.557219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2115.566768] Lustre: Skipped 2 previous similar messages [ 2116.364313] Lustre: server umount lustre-OST0000 complete [ 2120.673602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.675262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.684463] Lustre: Skipped 1 previous similar message [ 2120.700439] Lustre: Skipped 1 previous similar message [ 2126.970887] Lustre: server umount lustre-MDT0000 complete [ 2131.071765] LustreError: 57616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078821 with bad export cookie 11824390800515394433 [ 2131.074991] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2131.104856] LustreError: 57616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2131.492163] Lustre: server umount lustre-MDT0001 complete [ 2146.008829] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2149.472038] Key type lgssc unregistered [ 2150.010388] LNet: 61955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.034645] LNetError: 61955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.080176] LNet: Removed LNI 192.168.204.101@tcp [ 2151.096288] Key type .llcrypt unregistered [ 2151.098159] Key type ._llcrypt unregistered [ 2199.913957] Key type ._llcrypt registered [ 2199.922697] Key type .llcrypt registered [ 2200.176887] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2202.840514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2202.954057] alg: No test for adler32 (adler32-zlib) [ 2204.296128] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2204.614479] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2206.447463] Key type lgssc registered [ 2208.385778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2212.400767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2212.411119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.758266] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2213.809569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2214.018067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.793993] Lustre: Failing over lustre-MDT0000 [ 2216.466918] Lustre: server umount lustre-MDT0000 complete [ 2224.061752] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:15:13 (1785078913) [ 2231.250152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.624711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2235.861649] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2245.660699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.774108] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2245.795424] Lustre: 64755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2246.000818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2251.904431] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2261.031787] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2267.718935] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2276.224873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2276.423529] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2276.613971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.364964] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2284.772551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2291.820176] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2298.556446] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2307.553444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2307.565631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.584401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.624189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.635496] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.658209] Lustre: Skipped 1 previous similar message [ 2312.267224] Lustre: server umount lustre-MDT0000 complete [ 2316.768299] LustreError: 64745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079007 with bad export cookie 10343862734372507656 [ 2316.771287] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.781213] LustreError: 64745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2317.220472] Lustre: server umount lustre-MDT0001 complete [ 2321.312885] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2327.835975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2337.247441] Lustre: 63676:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079011/real 1785079011] req@000000005b81be58 x1871790886398336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2337.270991] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.283873] Lustre: 63674:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079012/real 1785079012] req@00000000df05645a x1871790886398464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2338.381963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.432583] LustreError: 63673:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000049b709e4 x1871790886399488/t0(0) o250->MGC192.168.204.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2342.434569] Lustre: 63674:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079016/real 1785079016] req@0000000014e34237 x1871790886398592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079032 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2343.017303] LustreError: 66606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.037618] LustreError: 66606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2343.163331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.292086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2346.145789] LustreError: 66607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.173814] LustreError: 66606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.192042] LustreError: 66606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2348.319483] Lustre: 63677:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079022/real 1785079022] req@000000007f55ca62 x1871790886398976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079038 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2348.349787] Lustre: 63677:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2348.878673] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2352.615717] LustreError: 68527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.633416] Lustre: 63675:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079027/real 1785079027] req@000000003bb93cdb x1871790886399104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079043 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2357.745921] LustreError: 66606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.789591] LustreError: 66606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2359.698397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.037528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2365.424600] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2366.519118] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2366.616221] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2376.455136] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2382.678803] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2391.014475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2391.023862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.038782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2391.041376] Lustre: Skipped 2 previous similar messages [ 2395.765754] Lustre: server umount lustre-MDT0000 complete [ 2397.161698] LustreError: 69454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.211087] LustreError: 69454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2400.118156] LustreError: 64745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079090 with bad export cookie 10343862734372509931 [ 2400.127203] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2400.134759] LustreError: 64745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2400.662240] Lustre: server umount lustre-MDT0001 complete [ 2407.737923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2418.107960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.464913] Lustre: 63676:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079092/real 1785079092] req@0000000030d5c2b3 x1871790886430080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2418.500911] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.520286] Lustre: Skipped 2 previous similar messages [ 2425.704575] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f8cc85a4fac9e4e [ 2425.718795] Lustre: MGC192.168.204.101@tcp: Connection restored to 0@lo (at 0@lo) [ 2426.138857] LustreError: 66607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.237689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.350340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2431.908472] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2444.117289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.773970] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2449.957069] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2459.523796] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2466.127283] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2475.490217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2475.510540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.549227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2475.559808] Lustre: Skipped 3 previous similar messages [ 2481.040926] Lustre: server umount lustre-MDT0000 complete [ 2482.656434] LustreError: 66607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.679628] LustreError: 66607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2486.203597] LustreError: 65490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079176 with bad export cookie 10343862734372511310 [ 2486.214406] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.234419] LustreError: 65490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2486.869712] Lustre: server umount lustre-MDT0001 complete [ 2495.704147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2502.176824] Lustre: DEBUG MARKER: files compared the same [ 2503.904262] Lustre: 63677:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079178/real 1785079178] req@00000000b50f2bf4 x1871790886459520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079194 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2503.971437] Lustre: 63677:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2503.998933] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.029824] Lustre: Skipped 2 previous similar messages [ 2506.509791] Lustre: server umount lustre-OST0000 complete [ 2530.644035] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2533.594212] Key type lgssc unregistered [ 2533.926810] LNet: 74487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2533.936051] LNetError: 74487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2533.958600] LNet: Removed LNI 192.168.204.101@tcp [ 2534.908608] Key type .llcrypt unregistered [ 2534.911163] Key type ._llcrypt unregistered [ 2560.115485] Key type ._llcrypt registered [ 2560.119377] Key type .llcrypt registered [ 2560.222818] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2561.479573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2561.677864] alg: No test for adler32 (adler32-zlib) [ 2562.792429] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2563.035716] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2564.695186] Key type lgssc registered [ 2565.811862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2574.452078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2574.493836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.862633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.240761] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug +malloc all [ 2586.084341] LustreError: 75930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.117091] LustreError: 75930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2591.102194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.204876] LustreError: 75931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.565750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2597.367947] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug +malloc all [ 2607.886773] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2615.030716] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2622.388598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2622.741649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.717786] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug +malloc all [ 2640.331707] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2646.320219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2656.742918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.745513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2656.754529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2656.776320] Lustre: Skipped 1 previous similar message [ 2660.855490] Lustre: server umount lustre-OST0000 complete [ 2666.996279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.996887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2667.023587] Lustre: Skipped 2 previous similar messages [ 2671.667330] Lustre: server umount lustre-MDT0000 complete [ 2672.097648] LustreError: 75930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.128112] LustreError: 75930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2676.072751] LustreError: 75912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079366 with bad export cookie 5011566291216636396 [ 2676.102652] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2676.570486] Lustre: server umount lustre-MDT0001 complete [ 2691.001490] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2694.204454] Key type lgssc unregistered [ 2694.517410] LNet: 79487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2694.521631] LNetError: 79487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2694.558065] LNet: Removed LNI 192.168.204.101@tcp [ 2695.442502] Key type .llcrypt unregistered [ 2695.446149] Key type ._llcrypt unregistered [ 2721.706359] Key type ._llcrypt registered [ 2721.709567] Key type .llcrypt registered [ 2721.822759] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2722.921100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2722.949432] alg: No test for adler32 (adler32-zlib) [ 2724.130251] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2724.430136] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2726.135946] Key type lgssc registered [ 2727.777600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2736.889745] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2736.917560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.415421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2757.535563] LustreError: 80879:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 2784.224139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.702089] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2800.006800] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2807.606402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.876094] Lustre: *** cfs_fail_loc=706, val=0*** [ 2808.258355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.370265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2813.406758] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2813.417441] LustreError: 80905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.445935] LustreError: 80905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2818.537935] LustreError: 80904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.560745] LustreError: 80905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.579764] LustreError: 80905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2822.586401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.081367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2828.202587] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2836.693855] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2844.213558] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2848.741298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2848.743738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.746094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.792928] Lustre: Skipped 1 previous similar message [ 2853.271236] Lustre: server umount lustre-OST0000 complete [ 2858.979025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.983467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2859.004660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2859.013719] Lustre: Skipped 2 previous similar messages [ 2863.842632] Lustre: server umount lustre-MDT0000 complete [ 2864.098264] LustreError: 81962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.127616] LustreError: 81962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2868.066636] LustreError: 80886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079558 with bad export cookie 16973657033766382350 [ 2868.077954] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2868.083979] LustreError: 80886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2868.518545] Lustre: server umount lustre-MDT0001 complete [ 2881.265785] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2884.731494] Key type lgssc unregistered [ 2885.195618] LNet: 84496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2885.209578] LNetError: 84496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2885.228957] LNet: Removed LNI 192.168.204.101@tcp [ 2886.435520] Key type .llcrypt unregistered [ 2886.437988] Key type ._llcrypt unregistered [ 2914.654844] Key type ._llcrypt registered [ 2914.657786] Key type .llcrypt registered [ 2914.753615] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2915.747475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2915.869374] alg: No test for adler32 (adler32-zlib) [ 2916.856608] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2917.041286] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2918.727227] Key type lgssc registered [ 2919.791654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2927.259717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2927.298571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.447292] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2940.873231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.482060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.112589] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2946.539259] LustreError: 86602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.564584] LustreError: 86602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2951.658031] LustreError: 86601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.691194] LustreError: 86601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2955.527627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2955.868436] LustreError: 86602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2955.883093] LustreError: 86602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2955.915980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.392708] LustreError: 86606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.430023] LustreError: 86606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2962.490816] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2966.508592] LustreError: 86605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.531286] LustreError: 86605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2970.780044] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2970.791929] Lustre: MGS: Client b6be6bec-ef55-49fb-8ec9-8daa8ece679d (at 0@lo) reconnecting [ 2970.826064] LustreError: 86616:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2971.077100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2971.172551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2975.859314] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2991.588976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.589653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2991.596805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2991.609105] Lustre: Skipped 1 previous similar message [ 2995.240712] Lustre: server umount lustre-OST0000 complete [ 3001.828164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.829584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3001.854215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3001.869680] Lustre: Skipped 2 previous similar messages [ 3006.313147] Lustre: server umount lustre-MDT0000 complete [ 3006.952159] LustreError: 86606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.967422] LustreError: 86606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3010.467658] Lustre: server umount lustre-MDT0001 complete [ 3014.253487] Lustre: server umount MGS complete [ 3026.783111] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:28:36 (1785079716) [ 3059.342064] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 3070.017888] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3110.933834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.771684] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3126.604684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.989122] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3127.139860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3127.273722] Lustre: lustre-MDT0001: new disk, initializing [ 3127.354723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.397968] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3127.409060] Lustre: 93772:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3127.421855] LustreError: 93772:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3132.220773] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3144.098289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3144.424591] Lustre: lustre-OST0000: new disk, initializing [ 3144.428324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3144.435370] Lustre: 94538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3144.519669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.237985] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3154.466947] Lustre: 94963:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000c76cda37 x1871791634030720/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785079860 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3162.381276] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3170.156417] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.174414] Lustre: MGS: Client dec9c57f-1970-4c8d-b226-c87bded4b2d4 (at 0@lo) reconnecting [ 3170.240623] Lustre: 93039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3170.265171] LustreError: 93758:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3170.339255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3170.393524] Lustre: lustre-MDT0000: new disk, initializing [ 3170.518277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.556786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3170.597570] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3171.681867] Lustre: 95609:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002feb6e63 x1871791634037632/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1785079877 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3175.133556] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3176.815633] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3180.854761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3180.862466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3180.913808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3206.628240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3206.652329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.672095] Lustre: Skipped 1 previous similar message [ 3206.675582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.208453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.220431] Lustre: Skipped 2 previous similar messages [ 3211.781158] Lustre: server umount lustre-OST0000 complete [ 3220.453199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3220.456795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.469546] LustreError: Skipped 1 previous similar message [ 3220.496405] Lustre: Skipped 1 previous similar message [ 3220.497652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3222.565438] Lustre: server umount lustre-MDT0000 complete [ 3225.577500] LustreError: 93742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.611420] LustreError: 93742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3233.905300] Lustre: server umount lustre-MDT0001 complete [ 3238.342828] Lustre: server umount MGS complete [ 3250.403881] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:32:20 (1785079940) [ 3270.375707] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3273.451883] Key type lgssc unregistered [ 3273.781265] LNet: 98110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3273.790741] LNetError: 98110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3273.818340] LNet: Removed LNI 192.168.204.101@tcp [ 3275.137542] Key type .llcrypt unregistered [ 3275.140879] Key type ._llcrypt unregistered [ 3295.714492] Key type ._llcrypt registered [ 3295.718485] Key type .llcrypt registered [ 3295.807443] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3296.943370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3297.081702] alg: No test for adler32 (adler32-zlib) [ 3298.255844] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3298.541638] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3300.209128] Key type lgssc registered [ 3301.519121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3309.412699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3309.423730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.941209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.724791] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3321.316314] LustreError: 99347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.342808] LustreError: 99347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3324.819197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.167914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3330.066203] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3337.512693] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3344.519071] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3352.299812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.931728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.041805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3360.380492] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3369.247269] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3380.715871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3380.724834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3380.737439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3380.739840] Lustre: Skipped 1 previous similar message [ 3380.775698] Lustre: Skipped 1 previous similar message [ 3385.017377] Lustre: server umount lustre-OST0000 complete [ 3390.963263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.967074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3390.996900] Lustre: Skipped 1 previous similar message [ 3395.678607] Lustre: server umount lustre-MDT0000 complete [ 3396.068717] LustreError: 101572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.082270] LustreError: 101572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3399.779428] LustreError: 99328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080090 with bad export cookie 1808174194778579181 [ 3399.788201] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3400.107380] Lustre: server umount lustre-MDT0001 complete [ 3413.870098] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3418.095962] Key type lgssc unregistered [ 3418.561299] LNet: 102882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3418.569577] LNetError: 102882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3418.592206] LNet: Removed LNI 192.168.204.101@tcp [ 3419.423513] Key type .llcrypt unregistered [ 3419.427795] Key type ._llcrypt unregistered [ 3439.309276] Key type ._llcrypt registered [ 3439.311195] Key type .llcrypt registered [ 3439.504057] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3440.723256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3440.844228] alg: No test for adler32 (adler32-zlib) [ 3441.848614] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3442.115071] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3443.874948] Key type lgssc registered [ 3445.334349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3452.506652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3452.512720] LustreError: 103962:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3452.581341] LustreError: 103966:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3452.592480] LustreError: 103962:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4926 [ 3452.597330] LustreError: 103962:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3452.612759] LustreError: 103962:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3452.625519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.169691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3463.348184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.734182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3469.316477] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3476.562539] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing lsmod [ 3479.839717] LustreError: 105173:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3479.850504] LustreError: 105170:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3479.859338] LustreError: 105173:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3479.869102] LustreError: 105173:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3479.881695] LustreError: 105173:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3479.897223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.376785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.130362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3484.138380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3484.148744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.667795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.682766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.701318] Lustre: Skipped 1 previous similar message [ 3487.781672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3487.798124] Lustre: Skipped 4 previous similar messages [ 3490.454892] Lustre: server umount lustre-MDT0000 complete [ 3492.838653] LustreError: 104004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3492.874292] LustreError: 104004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3495.226905] LustreError: 103978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080185 with bad export cookie 1853989621579280578 [ 3495.261039] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3496.031728] LustreError: 104436:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009706db33 x1871792184514560/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3496.393247] Lustre: server umount lustre-MDT0001 complete [ 3511.824592] Lustre: server umount lustre-OST0000 complete [ 3522.501686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.167208] LustreError: 106241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.236801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.681760] LustreError: 106242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3528.952768] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3532.770460] LustreError: 106241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.891294] LustreError: 106242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.810570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.167113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.516222] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3553.142648] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3560.116040] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3567.410957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3567.854971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3568.886989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3574.313714] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3583.582214] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3594.732019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.747421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3594.769086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.805429] LustreError: Skipped 1 previous similar message [ 3597.101258] Lustre: server umount lustre-OST0000 complete [ 3604.962960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.973929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.985484] Lustre: Skipped 2 previous similar messages [ 3604.996243] Lustre: Skipped 1 previous similar message [ 3607.824175] Lustre: server umount lustre-MDT0000 complete [ 3610.081432] LustreError: 107400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.100048] LustreError: 107400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3613.690074] LustreError: 106221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080303 with bad export cookie 1853989621579281278 [ 3613.699925] LustreError: 106221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3613.702548] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3614.192693] Lustre: server umount lustre-MDT0001 complete [ 3628.424392] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3632.428652] Key type lgssc unregistered [ 3632.816369] LNet: 109782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3632.831295] LNetError: 109782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3632.849468] LNet: Removed LNI 192.168.204.101@tcp [ 3634.007896] Key type .llcrypt unregistered [ 3634.010176] Key type ._llcrypt unregistered [ 3662.069791] Key type ._llcrypt registered [ 3662.077726] Key type .llcrypt registered [ 3662.197545] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3663.890818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3663.933990] alg: No test for adler32 (adler32-zlib) [ 3665.180967] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3665.493127] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3667.215258] Key type lgssc registered [ 3669.156849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3678.749685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3678.774820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.286760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3685.935209] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3690.471666] LustreError: 111205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3690.487803] LustreError: 111205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3695.590519] LustreError: 111204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3696.465183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3696.869335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3702.475923] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3711.456578] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3719.205579] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3727.403864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3727.919079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.242744] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3748.293174] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3750.465077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3761.837779] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3768.002702] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3775.977590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.979178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3775.987298] Lustre: Skipped 1 previous similar message [ 3775.997328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3778.765564] Lustre: server umount lustre-OST0000 complete [ 3784.161088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3784.173668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.210712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3784.221979] Lustre: Skipped 1 previous similar message [ 3786.213429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3786.214823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.385890] Lustre: server umount lustre-MDT0000 complete [ 3791.331280] LustreError: 111205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.354519] LustreError: 111205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3794.075245] LustreError: 111184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080484 with bad export cookie 18206417324821777311 [ 3794.080135] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.090185] LustreError: 111184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3794.568869] Lustre: server umount lustre-MDT0001 complete [ 3808.270983] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3811.764406] Key type lgssc unregistered [ 3812.130703] LNet: 114869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3812.142317] LNetError: 114869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3813.160760] LNet: Removed LNI 192.168.204.101@tcp [ 3814.431500] Key type .llcrypt unregistered [ 3814.436079] Key type ._llcrypt unregistered [ 3835.761185] Key type ._llcrypt registered [ 3835.764990] Key type .llcrypt registered [ 3835.901586] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3837.568254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3837.657672] alg: No test for adler32 (adler32-zlib) [ 3838.675813] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3838.911810] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3840.607211] Key type lgssc registered [ 3842.197948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3852.243540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3852.272600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.726788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3859.408585] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3863.014861] LustreError: 116091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.035879] LustreError: 116091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3868.131511] LustreError: 116092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.120543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.628127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3877.395104] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3886.251503] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3893.878801] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3903.762584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.173857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.634554] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3923.285976] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3923.687396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3934.183778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3934.188800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.193064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3934.210695] Lustre: Skipped 1 previous similar message [ 3939.308147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3939.319379] Lustre: Skipped 2 previous similar messages [ 3939.926501] Lustre: server umount lustre-OST0000 complete [ 3944.416860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.419373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3944.442616] Lustre: Skipped 1 previous similar message [ 3944.459276] Lustre: Skipped 1 previous similar message [ 3949.539261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.551690] Lustre: Skipped 1 previous similar message [ 3950.610960] Lustre: server umount lustre-MDT0000 complete [ 3954.660704] LustreError: 116091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3954.683730] LustreError: 116091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3955.566522] LustreError: 116073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080645 with bad export cookie 9914797156508402637 [ 3955.582543] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3955.583372] LustreError: 116073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3956.059503] Lustre: server umount lustre-MDT0001 complete [ 3971.800701] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3975.010966] Key type lgssc unregistered [ 3975.390066] LNet: 119627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3975.400902] LNetError: 119627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3975.432948] LNet: Removed LNI 192.168.204.101@tcp [ 3976.515633] Key type .llcrypt unregistered [ 3976.519180] Key type ._llcrypt unregistered [ 4007.136651] Key type ._llcrypt registered [ 4007.141564] Key type .llcrypt registered [ 4007.276953] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4008.733344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4008.939455] alg: No test for adler32 (adler32-zlib) [ 4010.265422] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4010.590774] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4012.335519] Key type lgssc registered [ 4013.667037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4023.189030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4023.233144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4024.999812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.559447] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4035.555978] LustreError: 121053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.586333] LustreError: 121053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4040.128905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.599422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4045.537539] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4054.935775] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4061.980896] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4070.074443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4070.376194] LustreError: 122875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.689948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.052252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 4079.473912] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4089.494658] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4097.743532] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4102.006379] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4102.013985] Lustre: Skipped 2 previous similar messages [ 4106.386386] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4106.417437] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4107.454602] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4107.463642] Lustre: Skipped 2 previous similar messages [ 4113.017315] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4113.027456] Lustre: Skipped 1 previous similar message [ 4118.237633] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.1@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4118.249471] Lustre: Skipped 2 previous similar messages [ 4121.516964] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.1@tcp 192.168.0.[2,10]@tcp [ 4132.694533] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.1/24@tcp in log lustre-MDT0000 [ 4132.707701] Lustre: Skipped 2 previous similar messages [ 4139.181935] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.1/24@tcp [ 4139.198575] Lustre: Skipped 1 previous similar message [ 4151.803827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.823504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4151.847754] Lustre: Skipped 1 previous similar message [ 4151.858629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4151.883768] Lustre: Skipped 1 previous similar message [ 4154.300154] Lustre: server umount lustre-OST0000 complete [ 4162.026119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.050618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4162.062526] Lustre: Skipped 1 previous similar message [ 4166.101168] Lustre: server umount lustre-MDT0000 complete [ 4167.136742] LustreError: 121054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.181877] LustreError: 121054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4171.186585] LustreError: 121034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080861 with bad export cookie 6225113331035157014 [ 4171.188410] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4171.216182] LustreError: 121034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4171.958978] Lustre: server umount lustre-MDT0001 complete [ 4186.650742] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4190.148268] Key type lgssc unregistered [ 4190.575672] LNet: 125905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4190.582701] LNetError: 125905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4190.599349] LNet: Removed LNI 192.168.204.101@tcp [ 4191.745505] Key type .llcrypt unregistered [ 4191.747919] Key type ._llcrypt unregistered [ 4226.619166] Key type ._llcrypt registered [ 4226.629821] Key type .llcrypt registered [ 4226.799905] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4228.782480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4228.826263] alg: No test for adler32 (adler32-zlib) [ 4230.079508] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4230.585477] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4232.295341] Key type lgssc registered [ 4234.306064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4244.758623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4244.797657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.555757] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4246.607795] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.1/24@tcp [ 4246.617735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.331706] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4256.747069] LustreError: 127468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.772409] LustreError: 127468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4261.857995] LustreError: 127469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.019114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4263.474144] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4263.555426] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.1/24@tcp [ 4263.598186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4268.840919] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4277.564054] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4285.193080] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4293.263614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4293.683843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4299.793769] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4308.450867] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4317.251449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4332.513148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4332.518310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.518958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4336.792493] Lustre: server umount lustre-OST0000 complete [ 4342.754122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.763452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4342.785243] Lustre: Skipped 2 previous similar messages [ 4342.797716] Lustre: Skipped 1 previous similar message [ 4347.874139] LustreError: 129598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4347.905438] LustreError: 129598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4347.933072] Lustre: server umount lustre-MDT0000 complete [ 4351.970399] LustreError: 127465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4351.988928] LustreError: 127465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4352.112822] LustreError: 127450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081042 with bad export cookie 13756399136338906859 [ 4352.122214] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4352.126506] LustreError: 127450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4352.683502] Lustre: server umount lustre-MDT0001 complete [ 4364.485047] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4367.580996] Key type lgssc unregistered [ 4367.981624] LNet: 131080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4367.986085] LNetError: 131080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4368.014496] LNet: Removed LNI 192.168.204.101@tcp [ 4368.891650] Key type .llcrypt unregistered [ 4368.899740] Key type ._llcrypt unregistered [ 4396.183335] Key type ._llcrypt registered [ 4396.189598] Key type .llcrypt registered [ 4396.285938] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 4411.377581] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4412.796985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4412.962538] alg: No test for adler32 (adler32-zlib) [ 4414.023988] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4414.251608] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4415.903150] Key type lgssc registered [ 4417.198913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4455.298620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4455.335769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.662506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4456.855836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4457.028569] Lustre: lustre-MDT0000: new disk, initializing [ 4457.163273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4457.177872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4462.614751] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4474.167669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4474.313593] Lustre: 134482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4474.438208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4474.442677] Lustre: Skipped 1 previous similar message [ 4474.563183] Lustre: lustre-MDT0001: new disk, initializing [ 4474.690577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4474.736475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4474.750981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4480.206643] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4491.003130] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4497.920856] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4505.896074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4506.309764] Lustre: lustre-OST0000: new disk, initializing [ 4506.317092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4506.325409] Lustre: 136448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4506.459202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4513.017212] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4513.865710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4513.872544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4513.955764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4523.382908] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4528.527218] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4528.806043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4533.338541] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4533.528426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4539.874586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.880406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4539.895782] Lustre: Skipped 1 previous similar message [ 4539.912064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4542.095175] Lustre: server umount lustre-OST0000 complete [ 4546.535468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4546.546818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4546.571381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4546.580586] Lustre: Skipped 1 previous similar message [ 4550.112456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.114716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.466947] Lustre: server umount lustre-MDT0000 complete [ 4556.498656] LustreError: 136447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081246 with bad export cookie 5061533268324747122 [ 4556.500170] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4556.512233] LustreError: 136447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4556.757836] Lustre: server umount lustre-MDT0001 complete [ 4566.357758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.185859] LustreError: 138668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.210058] LustreError: 138668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4567.333363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4572.640740] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4572.642765] LustreError: 138669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.765307] LustreError: 138668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.330484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4581.762471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4586.633481] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4595.283284] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4602.705306] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4610.793408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4611.262827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4619.035951] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4627.823697] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4634.825938] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4635.074978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4641.728750] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4641.995502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4654.042819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.428722] Lustre: lustre-OST0001: new disk, initializing [ 4654.434411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4654.460120] Lustre: 142043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4654.581992] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4663.180915] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4663.995217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4664.004600] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4664.157045] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4674.938636] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4675.281517] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4681.154965] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4681.520780] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4688.749786] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b53031f9000.ost_server_uuid 50 [ 4690.299186] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b53031f9000.ost_server_uuid in IDLE state after 0 sec [ 4700.133655] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.136029] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4700.148432] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4700.148443] Lustre: Skipped 1 previous similar message [ 4700.152156] Lustre: Skipped 1 previous similar message [ 4700.183711] LustreError: Skipped 1 previous similar message [ 4704.811430] Lustre: server umount lustre-OST0001 complete [ 4705.273836] LustreError: 141480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.301371] LustreError: 141480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4710.373478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.386920] LustreError: 141480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.394255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4710.394270] Lustre: Skipped 1 previous similar message [ 4710.418503] LustreError: 141480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4720.611192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4720.612282] LustreError: 141480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.620139] Lustre: Skipped 4 previous similar messages [ 4720.635267] LustreError: 141480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4724.191928] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4724.593554] Lustre: server umount lustre-OST0000 complete [ 4730.336501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4730.348134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.383508] Lustre: Skipped 1 previous similar message [ 4735.663480] Lustre: server umount lustre-MDT0000 complete [ 4740.537647] LustreError: 138650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081430 with bad export cookie 5061533268324748004 [ 4740.547089] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4740.548250] LustreError: 138650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4740.954225] Lustre: server umount lustre-MDT0001 complete [ 4764.862943] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4767.832916] Key type lgssc unregistered [ 4768.214774] LNet: 145049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4768.223126] LNetError: 145049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4768.237850] LNet: Removed LNI 192.168.204.101@tcp [ 4769.103029] Key type .llcrypt unregistered [ 4769.105973] Key type ._llcrypt unregistered [ 4813.796178] Key type ._llcrypt registered [ 4813.799228] Key type .llcrypt registered [ 4813.944944] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 4830.036336] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4831.559146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4831.570260] alg: No test for adler32 (adler32-zlib) [ 4832.769655] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4833.012977] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4834.695420] Key type lgssc registered [ 4836.116595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4876.880643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4876.897810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4878.181668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4878.228518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4878.350169] Lustre: lustre-MDT0000: new disk, initializing [ 4878.484980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4878.512664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4883.251788] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4894.422403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4894.516188] Lustre: 149206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4894.575064] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4894.579127] Lustre: Skipped 1 previous similar message [ 4894.646601] Lustre: lustre-MDT0001: new disk, initializing [ 4894.724873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4894.777103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4894.788375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4899.394450] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4908.486274] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4914.018719] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4921.677443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4921.984262] Lustre: lustre-OST0000: new disk, initializing [ 4921.988178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4922.001214] Lustre: 151174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4922.166064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4928.658671] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4929.617870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4929.629728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4929.748724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4938.216805] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4942.135389] Lustre: Failing over lustre-OST0000 [ 4942.285883] Lustre: server umount lustre-OST0000 complete [ 4960.738668] Lustre: 147427:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785081634/real 1785081634] req@00000000d4ce709b x1871793643095296/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1785081650 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4960.776596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.743260] Lustre: 147430:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785081635/real 1785081635] req@0000000010c4224a x1871793643095680/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785081651 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4961.779476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.756702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4963.281585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4963.312935] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4966.303131] Lustre: 147429:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785081640/real 1785081640] req@0000000001860c2d x1871793643097216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785081656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4966.357271] Lustre: 147429:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4969.926657] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4971.487294] Lustre: 147427:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785081645/real 1785081645] req@00000000f3e1ddf7 x1871793643098496/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785081661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4971.509775] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4972.618597] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4972.621462] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4975.323270] Lustre: Failing over lustre-MDT0000 [ 4975.772073] Lustre: server umount lustre-MDT0000 complete [ 4976.351152] Lustre: 147428:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785081650/real 1785081650] req@000000000983160a x1871793643099520/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785081666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4976.402347] Lustre: 147428:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4976.608236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.633716] Lustre: Skipped 1 previous similar message [ 4981.730751] LustreError: 150017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.786148] LustreError: 150017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4986.850135] LustreError: 149217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4986.887158] LustreError: 149217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4991.842708] Lustre: 147430:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785081666/real 1785081666] req@000000008c7e6c01 x1871793643111040/t0(0) o400->MGC192.168.204.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785081682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4991.914371] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4991.947467] LustreError: 149216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4991.984332] LustreError: 149216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4997.077172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4997.093853] LustreError: 149212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4997.113704] LustreError: 149212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5002.210277] LustreError: 149216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5002.226433] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2e902d104fc2510e [ 5002.245291] LustreError: 149216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5002.283182] Lustre: MGC192.168.204.101@tcp: Connection restored to 0@lo (at 0@lo) [ 5002.290165] Lustre: Skipped 1 previous similar message [ 5002.622519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.734952] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5007.342755] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5007.861795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5007.887205] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5007.932201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5008.190957] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5018.082123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.091405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5018.107470] Lustre: Skipped 2 previous similar messages [ 5018.118190] Lustre: Skipped 1 previous similar message [ 5022.546045] Lustre: server umount lustre-OST0000 complete [ 5028.329810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5028.334149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.343207] Lustre: Skipped 1 previous similar message [ 5028.345110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5028.351375] Lustre: Skipped 1 previous similar message [ 5032.962061] Lustre: server umount lustre-MDT0000 complete [ 5033.443098] LustreError: 151181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.471077] LustreError: 151181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5037.521084] LustreError: 149198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081727 with bad export cookie 3355231270471880974 [ 5037.523637] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5037.536981] LustreError: 149198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5038.488866] Lustre: server umount lustre-MDT0001 complete [ 5051.472516] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5055.533495] Key type lgssc unregistered [ 5055.894752] LNet: 154847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5055.903732] LNetError: 154847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5055.935730] LNet: Removed LNI 192.168.204.101@tcp [ 5056.940280] Key type .llcrypt unregistered [ 5056.944214] Key type ._llcrypt unregistered [ 5094.858173] Key type ._llcrypt registered [ 5094.860486] Key type .llcrypt registered [ 5094.973557] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5096.212636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5096.230775] alg: No test for adler32 (adler32-zlib) [ 5097.268500] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5097.563237] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5099.319182] Key type lgssc registered [ 5100.705848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5108.490726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5108.510776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5110.028945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.931302] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5120.484232] LustreError: 156372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5120.504574] LustreError: 156372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5125.091594] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5125.605123] LustreError: 156373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.823753] LustreError: 156401:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5128.831545] LustreError: 156401:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5128.836797] LustreError: 156401:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 5129.153572] Lustre: server umount lustre-MDT0000 complete [ 5144.756866] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 5156.918675] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5195.960219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.211543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5196.251224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5196.356374] Lustre: lustre-MDT0000: new disk, initializing [ 5196.493953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5196.519639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5201.481031] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5212.260578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5212.418697] Lustre: 159706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5212.468116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5212.476105] Lustre: Skipped 1 previous similar message [ 5212.565823] Lustre: lustre-MDT0001: new disk, initializing [ 5212.663095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5212.728141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5212.742071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5218.653586] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5230.165960] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5235.795534] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5243.596518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.914085] Lustre: lustre-OST0000: new disk, initializing [ 5243.917556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5243.923359] Lustre: 161674:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5244.053511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5250.643310] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5250.693812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5250.715419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5250.843159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5260.341574] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5265.139448] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5267.038453] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:05:56 (1785081956) [ 5537.761616] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5537.762919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.776113] Lustre: Skipped 2 previous similar messages [ 5537.796660] Lustre: Skipped 2 previous similar messages [ 5541.870556] Lustre: server umount lustre-MDT0000 complete [ 5542.880383] LustreError: 159718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.907544] LustreError: 159718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5545.315931] LustreError: 159699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082235 with bad export cookie 17610695919615707741 [ 5545.318716] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5545.325268] LustreError: 159699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5547.999988] LustreError: 163461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5548.005896] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5548.045373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5551.010443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5551.691706] Lustre: server umount lustre-MDT0001 complete [ 5562.380357] Lustre: server umount lustre-OST0000 complete [ 5572.315674] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 5583.857337] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5624.016589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5624.310975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5624.333317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5624.427090] Lustre: lustre-MDT0000: new disk, initializing [ 5624.506538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5624.548412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5628.933372] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5640.792753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.934531] Lustre: 166340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5640.983974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5640.987869] Lustre: Skipped 1 previous similar message [ 5641.061672] Lustre: lustre-MDT0001: new disk, initializing [ 5641.167510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5641.222253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5641.233767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5645.622870] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5655.413910] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5661.258037] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5668.687350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5669.035831] Lustre: lustre-OST0000: new disk, initializing [ 5669.043526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5669.055578] Lustre: 168309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5669.171622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5674.533353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5674.545441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5674.682618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5675.489596] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5685.175936] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5689.739511] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5689.966599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5694.661381] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5694.968735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5700.576262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5700.581222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5700.586779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5700.601217] Lustre: Skipped 1 previous similar message [ 5700.617265] Lustre: Skipped 1 previous similar message [ 5705.699461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.710363] Lustre: Skipped 1 previous similar message [ 5711.327150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5711.556884] Lustre: server umount lustre-OST0000 complete [ 5717.984914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5717.992618] LustreError: Skipped 1 previous similar message [ 5717.998409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.019364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.025652] Lustre: Skipped 2 previous similar messages [ 5722.808096] Lustre: server umount lustre-MDT0000 complete [ 5726.191259] LustreError: 166350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5726.218020] LustreError: 166350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5727.006028] LustreError: 166332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082417 with bad export cookie 17610695919615866571 [ 5727.012826] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5727.021911] LustreError: 166332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5727.397917] Lustre: server umount lustre-MDT0001 complete [ 5737.283782] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:13:46 (1785082426) [ 5756.207862] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 5766.752327] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5802.278685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.457915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5802.484924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5802.581838] Lustre: lustre-MDT0000: new disk, initializing [ 5802.665915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.680246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5806.710833] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5816.906728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.023793] Lustre: 173067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5817.264623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5821.469482] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5829.407585] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5835.326651] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5842.731912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5842.990596] Lustre: lustre-OST0000: new disk, initializing [ 5842.992700] Lustre: Skipped 1 previous similar message [ 5842.997734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5843.001595] Lustre: Skipped 2 previous similar messages [ 5843.005481] Lustre: 175044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5849.752246] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5852.739163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5852.760284] Lustre: Skipped 1 previous similar message [ 5852.775559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5852.991729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5860.558561] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5868.529425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.540765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.548519] Lustre: Skipped 2 previous similar messages [ 5868.548764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.548768] Lustre: Skipped 2 previous similar messages [ 5873.653460] LustreError: 175025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.689529] LustreError: 175025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5873.785576] Lustre: server umount lustre-OST0000 complete [ 5878.754106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5878.761883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.797862] Lustre: Skipped 1 previous similar message [ 5884.691845] Lustre: server umount lustre-MDT0000 complete [ 5888.583158] LustreError: 173056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082578 with bad export cookie 17610695919615867453 [ 5888.590738] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5888.595515] LustreError: 173056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5889.272150] Lustre: server umount lustre-MDT0001 complete [ 5903.435087] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:16:33 (1785082593) [ 5919.871470] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 5929.868576] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5966.129562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.352825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5966.375723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5966.445141] Lustre: lustre-MDT0000: new disk, initializing [ 5966.493612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.497522] Lustre: Skipped 2 previous similar messages [ 5966.507921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5970.737109] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5982.467443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.622791] Lustre: 179368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5983.056344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5988.923186] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5998.539222] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6005.167666] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6012.934191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6013.422247] Lustre: 181348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6021.194599] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6023.315061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6023.520198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6031.729707] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6044.138668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.140339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6044.145294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6044.145306] Lustre: Skipped 5 previous similar messages [ 6044.161678] Lustre: Skipped 1 previous similar message [ 6044.200492] LustreError: Skipped 1 previous similar message [ 6046.337957] Lustre: server umount lustre-OST0000 complete [ 6056.639873] Lustre: server umount lustre-MDT0000 complete [ 6059.491800] LustreError: 179380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6059.527183] LustreError: 179380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6061.322193] LustreError: 181346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082751 with bad export cookie 17610695919615868538 [ 6061.331661] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6061.339290] LustreError: 181346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6061.891203] Lustre: server umount lustre-MDT0001 complete [ 6074.115208] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 6077.064429] Key type lgssc unregistered [ 6077.361147] LNet: 183204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6077.374056] LNetError: 183204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6077.399949] LNet: Removed LNI 192.168.204.101@tcp [ 6078.549237] Key type .llcrypt unregistered [ 6078.551063] Key type ._llcrypt unregistered [ 6106.889431] Key type ._llcrypt registered [ 6106.896349] Key type .llcrypt registered [ 6107.092729] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 6108.394461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6108.489364] alg: No test for adler32 (adler32-zlib) [ 6109.695711] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6109.949877] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 6111.632149] Key type lgssc registered [ 6113.062786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6121.103204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6121.126332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.556534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6127.310356] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6132.706119] LustreError: 184625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6132.746540] LustreError: 184625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6136.553423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.052124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6143.147450] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6152.129888] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6158.189625] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6165.770354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6166.399618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6167.429231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6173.782795] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6182.352196] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6203.365803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6203.367656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6203.377529] LustreError: Skipped 1 previous similar message [ 6203.383903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6203.396454] Lustre: Skipped 1 previous similar message [ 6203.412814] Lustre: Skipped 1 previous similar message [ 6207.782868] Lustre: server umount lustre-OST0000 complete [ 6213.601656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6213.618425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6213.636342] Lustre: Skipped 1 previous similar message [ 6213.655806] Lustre: Skipped 1 previous similar message [ 6218.728230] LustreError: 184625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.744264] LustreError: 184625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6218.909808] Lustre: server umount lustre-MDT0000 complete [ 6222.559885] LustreError: 184607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785082912 with bad export cookie 6647986791130291818 [ 6222.568118] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6222.573736] LustreError: 184607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6222.926837] Lustre: server umount lustre-MDT0001 complete [ 6234.253303] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 6236.767605] Key type lgssc unregistered [ 6237.026788] LNet: 188178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6237.033908] LNetError: 188178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6238.053867] LNet: Removed LNI 192.168.204.101@tcp [ 6238.926668] Key type .llcrypt unregistered [ 6238.929797] Key type ._llcrypt unregistered [ 6262.396541] Key type ._llcrypt registered [ 6262.399826] Key type .llcrypt registered [ 6262.534409] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 6263.402284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6263.428667] alg: No test for adler32 (adler32-zlib) [ 6264.532629] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6264.801912] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 6266.535185] Key type lgssc registered [ 6267.653091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6274.734633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6274.762983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.296797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6280.499950] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6285.801868] LustreError: 189611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6285.824620] LustreError: 189611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6288.869206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.161194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6293.205959] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6300.570757] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6306.468680] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6313.268290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6313.662940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6319.184179] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6320.811464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6326.278745] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6333.451779] Lustre: lustre-OST0000: Not available for connect from 192.168.204.1@tcp (stopping) [ 6335.460833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6335.468317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.491388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6336.485441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6336.490846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.510113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6336.514030] Lustre: Skipped 1 previous similar message [ 6337.890604] Lustre: server umount lustre-OST0000 complete [ 6346.266159] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b53058d7800.ost_server_uuid 50 [ 6348.280554] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6366.178539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6366.180132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6366.201238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6366.215695] Lustre: Skipped 1 previous similar message [ 6366.233216] Lustre: Skipped 1 previous similar message [ 6369.196567] Lustre: server umount lustre-MDT0000 complete [ 6371.296579] LustreError: 190339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6371.315077] LustreError: 190339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6373.212034] LustreError: 189591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083063 with bad export cookie 1088238699129541305 [ 6373.229956] LustreError: 189591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6373.231800] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.712922] Lustre: server umount lustre-MDT0001 complete [ 6382.861445] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:24:32 (1785083072) [ 6390.964636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6391.432181] LustreError: 193524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6391.501684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.183585] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6396.901195] LustreError: 193525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6401.002372] LustreError: 193524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.573435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.028099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6410.503822] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6417.778838] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6423.628092] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6430.618501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6430.960652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6436.097196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6436.827585] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6445.359632] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6452.693868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6452.926649] Lustre: lustre-OST0001: new disk, initializing [ 6452.930286] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6452.936105] Lustre: 196367:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6453.032457] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6458.528865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6458.543788] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6458.641765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6460.712310] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6470.746618] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6474.215378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.216573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6474.233029] Lustre: Skipped 1 previous similar message [ 6474.251155] Lustre: Skipped 1 previous similar message [ 6480.207047] Lustre: server umount lustre-OST0000 complete [ 6481.894351] LustreError: 196034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.421173] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6488.773164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6494.409422] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6494.733117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6499.814406] LustreError: 195347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.848947] LustreError: 195347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6510.059612] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6510.062329] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.078452] LustreError: Skipped 1 previous similar message [ 6510.081434] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6510.081441] Lustre: Skipped 3 previous similar messages [ 6510.124592] Lustre: Skipped 1 previous similar message [ 6514.906522] Lustre: server umount lustre-OST0001 complete [ 6518.752525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6518.758890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6524.867869] Lustre: server umount lustre-MDT0000 complete [ 6529.304602] LustreError: 193505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083219 with bad export cookie 1088238699129542383 [ 6529.320797] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6529.779714] Lustre: server umount lustre-MDT0001 complete [ 6552.401442] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:27:22 (1785083242) [ 6558.850080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6559.148480] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6559.184871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6559.361920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.747133] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6572.085953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6572.154621] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6572.176688] Lustre: 199806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6577.280598] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6584.770127] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6591.641779] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6599.639524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6599.907758] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6607.439093] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6608.043182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6618.226889] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6626.363244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6626.592398] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6626.992523] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6627.003835] Lustre: Skipped 2 previous similar messages [ 6634.381127] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6643.129556] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6646.244828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6646.250431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6646.257074] Lustre: Skipped 2 previous similar messages [ 6646.273015] Lustre: Skipped 5 previous similar messages [ 6652.253682] Lustre: server umount lustre-OST0000 complete [ 6654.451876] LustreError: 201653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6654.478112] LustreError: 201653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6671.328854] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6674.789684] Lustre: server umount lustre-OST0001 complete [ 6680.034332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.059265] Lustre: Skipped 2 previous similar messages [ 6680.065946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6680.078312] Lustre: Skipped 5 previous similar messages [ 6685.979326] Lustre: server umount lustre-MDT0000 complete [ 6689.770945] LustreError: 199796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083380 with bad export cookie 1088238699129543783 [ 6689.777957] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6689.785886] LustreError: 199796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6690.025588] Lustre: server umount lustre-MDT0001 complete [ 6716.751410] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:30:06 (1785083406) [ 6734.098378] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 6743.845826] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 6783.338567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.624506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6783.668846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6783.748507] Lustre: lustre-MDT0000: new disk, initializing [ 6783.860154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.875923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6788.722552] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6801.000260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6801.142543] Lustre: 207905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6801.183967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6801.187584] Lustre: Skipped 1 previous similar message [ 6801.238277] Lustre: lustre-MDT0001: new disk, initializing [ 6801.335100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6801.351114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6806.475048] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6816.398403] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6822.653365] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6829.959418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.228746] Lustre: lustre-OST0000: new disk, initializing [ 6830.232197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6830.239236] Lustre: 209873:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6836.975576] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6838.914123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6838.925585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6839.048235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6848.045376] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6852.956838] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6853.169733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6858.573206] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6858.815103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6864.878729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6864.883591] LustreError: Skipped 2 previous similar messages [ 6864.888213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.897119] Lustre: Skipped 1 previous similar message [ 6864.903278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6864.913257] Lustre: Skipped 2 previous similar messages [ 6875.106860] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6875.289619] Lustre: server umount lustre-OST0000 complete [ 6885.124688] Lustre: server umount lustre-MDT0000 complete [ 6885.351244] LustreError: 207914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.383399] LustreError: 207914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6889.905951] LustreError: 207896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083580 with bad export cookie 1088238699129545351 [ 6889.932040] LustreError: 207896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6889.945779] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6890.541356] Lustre: server umount lustre-MDT0001 complete [ 6901.308568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.748232] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6916.150191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.928729] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6929.850821] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6936.349734] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6944.067744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6951.183067] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6959.301776] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6964.707748] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6964.909577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6970.609861] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6970.926393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6973.411717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6973.417803] LustreError: Skipped 1 previous similar message [ 6979.634234] Lustre: server umount lustre-OST0000 complete [ 6986.685809] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6991.537714] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7022.829411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7026.368363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7026.387406] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7030.383988] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7039.304620] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7040.759917] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7040.777298] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7041.120764] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7047.138727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7047.143062] LustreError: Skipped 1 previous similar message [ 7047.156810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7047.169373] Lustre: Skipped 4 previous similar messages [ 7047.174700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7047.178912] Lustre: Skipped 9 previous similar messages [ 7052.699248] Lustre: server umount lustre-OST0000 complete [ 7064.690791] Lustre: server umount lustre-MDT0000 complete [ 7066.592832] LustreError: 212090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7066.606302] LustreError: 212090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7069.529852] LustreError: 212073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785083759 with bad export cookie 1088238699129546226 [ 7069.535988] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7069.553987] LustreError: 212073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7070.174405] Lustre: server umount lustre-MDT0001 complete [ 7081.138337] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:36:10 (1785083770) [ 7090.278201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.403766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7091.409524] Lustre: Skipped 6 previous similar messages [ 7098.990427] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7110.532171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.229460] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7127.135196] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7133.847885] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7141.892469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7150.390655] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7158.934377] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7164.119694] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7164.321365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7169.761878] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7170.057317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7179.303423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7179.597287] Lustre: lustre-OST0001: new disk, initializing [ 7179.599933] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7179.604362] Lustre: 221435:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7185.592137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7185.611303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7185.778960] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7187.981486] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7200.553811] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7207.452619] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7207.727203] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7214.273634] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7214.487914] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7221.736494] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7221.747174] LustreError: Skipped 2 previous similar messages [ 7222.921940] Lustre: server umount lustre-OST0001 complete [ 7230.640788] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7230.961652] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7236.490336] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7236.750552] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7267.439357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7272.960910] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7272.978836] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7272.993465] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7273.019876] Lustre: Skipped 1 previous similar message [ 7275.101671] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7283.720806] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7285.413318] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7293.781154] Lustre: server umount lustre-OST0001 complete [ 7313.896872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.908325] Lustre: Skipped 7 previous similar messages [ 7313.920401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7313.928362] Lustre: Skipped 12 previous similar messages [ 7318.634659] Lustre: server umount lustre-OST0000 complete [ 7330.402282] Lustre: server umount lustre-MDT0000 complete [ 7334.373845] LustreError: 219904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7334.404217] LustreError: 219904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 35 previous similar messages [ 7335.770454] LustreError: 218800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084026 with bad export cookie 1088238699129547199 [ 7335.771718] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.798449] LustreError: 218800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7350.239465] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7350.647241] Lustre: server umount lustre-MDT0001 complete [ 7376.993195] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:41:06 (1785084066) [ 7384.683381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7384.948211] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7384.976223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7390.605500] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7400.660268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7400.853462] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7400.936958] Lustre: 226683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7406.593813] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7415.332758] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7422.579358] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7430.660824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7430.860371] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7438.653931] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7448.429282] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7460.335250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7460.448519] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7468.029638] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7476.521476] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7478.283250] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7484.448185] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7484.831881] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7491.170336] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7491.541586] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7499.028782] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b5306d17000.ost_server_uuid 50 [ 7500.799709] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b5306d17000.ost_server_uuid in IDLE state after 0 sec [ 7502.330158] Lustre: Permanently deactivating lustre-OST0001 [ 7502.378873] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7506.883502] Lustre: Permanently reactivating lustre-OST0001 [ 7506.910299] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7506.917583] Lustre: Skipped 2 previous similar messages [ 7513.064298] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7513.079597] LustreError: Skipped 3 previous similar messages [ 7517.878492] Lustre: server umount lustre-OST0001 complete [ 7529.778399] Lustre: server umount lustre-OST0000 complete [ 7541.897816] Lustre: server umount lustre-MDT0000 complete [ 7546.482694] LustreError: 227434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084236 with bad export cookie 1088238699129548487 [ 7546.501812] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7547.137134] Lustre: server umount lustre-MDT0001 complete [ 7579.129572] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:44:27 (1785084267) [ 7589.051278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.331280] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7589.371548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7589.378541] Lustre: Skipped 2 previous similar messages [ 7595.424199] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7605.249265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7605.394060] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7605.437600] Lustre: 233382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7605.854596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7605.864453] Lustre: Skipped 9 previous similar messages [ 7610.663558] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7619.332739] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7625.577498] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7633.673141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7633.889434] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7633.926484] Lustre: Permanently deactivating lustre-OST0000 [ 7633.955735] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7641.670138] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7643.710817] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7651.704874] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7660.928327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7661.011447] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7669.184393] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7670.480443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7676.873346] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7678.859222] Lustre: Permanently reactivating lustre-OST0000 [ 7678.875230] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7678.880843] Lustre: Skipped 2 previous similar messages [ 7679.671328] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7679.688095] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000a84ee85f, cur 1785084369 deadline 1785084364 last 1785084334 [ 7679.751106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7697.795963] Lustre: server umount lustre-OST0001 complete [ 7716.322035] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7716.537649] Lustre: server umount lustre-OST0000 complete [ 7726.892409] Lustre: server umount lustre-MDT0000 complete [ 7730.639521] LustreError: 233373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084420 with bad export cookie 1088238699129550132 [ 7730.642580] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7730.648708] LustreError: 233373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7731.068633] Lustre: server umount lustre-MDT0001 complete [ 7742.164757] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7745.080360] Key type lgssc unregistered [ 7745.416812] LNet: 238266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7745.426684] LNetError: 238266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7745.459096] LNet: Removed LNI 192.168.204.101@tcp [ 7746.349517] Key type .llcrypt unregistered [ 7746.351758] Key type ._llcrypt unregistered [ 7769.044829] Key type ._llcrypt registered [ 7769.048750] Key type .llcrypt registered [ 7769.159194] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7770.332164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7770.511527] alg: No test for adler32 (adler32-zlib) [ 7771.593837] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7771.872177] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7773.527197] Key type lgssc registered [ 7774.562578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7785.399837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7785.424559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.004236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7791.557593] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7797.233606] LustreError: 239727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7797.267972] LustreError: 239727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7800.795918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7800.881603] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7800.911886] Lustre: 239717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7800.924529] Lustre: Permanently deactivating lustre-MDT0001 [ 7800.939753] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7801.337714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7803.152598] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7803.158712] LustreError: 239758:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7803.181872] LustreError: 239758:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7803.201749] LustreError: 239758:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7806.267566] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7814.691669] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7822.410217] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7830.441374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7830.765240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7831.791589] LustreError: 241567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7831.837274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7837.169947] LustreError: 241567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.942912] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7842.282322] LustreError: 241567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.370200] LustreError: 241568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.387097] LustreError: 241568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7846.395410] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7846.611917] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7853.890630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.235078] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7859.709039] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7859.721316] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7860.983408] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7868.703350] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7872.891414] Lustre: Permanently reactivating lustre-MDT0001 [ 7872.911686] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7872.916864] Lustre: Skipped 2 previous similar messages [ 7874.250243] Lustre: lustre-MDT0001: Client eb608b78-dca3-484d-be21-ffd292f884f3 (at 192.168.204.1@tcp) reconnecting [ 7879.643852] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7881.267691] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7886.030649] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7886.201157] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7891.155344] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7891.375943] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7894.279438] Lustre: Permanently deactivating lustre-MDT0001 [ 7894.307402] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7894.312962] Lustre: Skipped 2 previous similar messages [ 7903.301440] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7913.958486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7913.971574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7913.988557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.003120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.007762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.017362] Lustre: Skipped 2 previous similar messages [ 7916.025790] Lustre: Skipped 3 previous similar messages [ 7918.264904] Lustre: server umount lustre-MDT0000 complete [ 7921.122316] LustreError: 239723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7921.150150] LustreError: 239723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7922.018714] LustreError: 244210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084612 with bad export cookie 71764158540639807 [ 7922.021019] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7922.036815] LustreError: 244210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7922.510385] Lustre: server umount lustre-MDT0001 complete [ 7928.538947] Lustre: server umount lustre-OST0000 complete [ 7941.339849] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:50:30 (1785084630) [ 7951.958726] Lustre: server umount lustre-OST0001 complete [ 7958.466511] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 7969.078764] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8002.905190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8003.129152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8003.139367] Lustre: Skipped 2 previous similar messages [ 8003.170981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8003.243848] Lustre: lustre-MDT0000: new disk, initializing [ 8003.353228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8003.379380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8007.117646] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8016.827578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8016.929745] Lustre: 248004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8016.987256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8016.994151] Lustre: Skipped 1 previous similar message [ 8017.082135] Lustre: lustre-MDT0001: new disk, initializing [ 8017.166798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8017.231737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8017.247931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8021.168878] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8029.446651] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8034.595989] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8040.790782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8041.019299] Lustre: lustre-OST0000: new disk, initializing [ 8041.022812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8041.028104] Lustre: 249975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8041.127422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.191542] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8048.204529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8048.218607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8048.338253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8055.381662] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8057.847128] LustreError: 248013:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8067.173705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8067.328330] Lustre: lustre-OST0001: new disk, initializing [ 8067.332649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8067.339952] Lustre: 251088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8067.905915] LustreError: 248013:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8067.926928] LustreError: 249376:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8072.939949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8072.967907] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8073.097908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8073.278479] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8077.991186] LustreError: 249376:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8084.199145] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8085.768874] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8088.554567] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8088.555893] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.567627] LustreError: Skipped 1 previous similar message [ 8088.577316] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8088.581994] Lustre: Skipped 1 previous similar message [ 8088.595270] Lustre: Skipped 1 previous similar message [ 8093.681655] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8093.696561] Lustre: Skipped 1 previous similar message [ 8093.801670] Lustre: server umount lustre-OST0001 complete [ 8098.789058] LustreError: 249958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8098.828733] LustreError: 249958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8114.153534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.158205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.190372] Lustre: Skipped 1 previous similar message [ 8114.228791] Lustre: Skipped 1 previous similar message [ 8117.190768] Lustre: server umount lustre-OST0000 complete [ 8124.385491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.418179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8128.494316] Lustre: server umount lustre-MDT0000 complete [ 8132.937983] LustreError: 247999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084823 with bad export cookie 71764158540642621 [ 8132.944831] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8132.947756] LustreError: 247999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8139.409058] Lustre: server umount lustre-MDT0001 complete [ 8151.103859] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8153.925764] Key type lgssc unregistered [ 8154.334621] LNet: 253120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8154.351684] LNetError: 253120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8154.370462] LNet: Removed LNI 192.168.204.101@tcp [ 8155.309694] Key type .llcrypt unregistered [ 8155.311361] Key type ._llcrypt unregistered [ 8193.528150] Key type ._llcrypt registered [ 8193.530577] Key type .llcrypt registered [ 8193.620544] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8194.831122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8194.942309] alg: No test for adler32 (adler32-zlib) [ 8195.976637] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8196.216514] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 8197.879476] Key type lgssc registered [ 8198.915573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8206.277730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8206.295746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8207.621641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8207.649381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8207.824723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8212.102933] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8221.077333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.166525] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8221.233348] Lustre: 255327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8221.565700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8225.905369] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8234.688817] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8240.646825] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8247.693872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8247.876557] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8248.077193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.220375] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8256.724031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8262.468574] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8317.922607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8317.941261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8317.967199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8318.438933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8318.450072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.473771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8318.482689] Lustre: Skipped 1 previous similar message [ 8320.052118] Lustre: server umount lustre-OST0000 complete [ 8324.699304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8337.944621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.328791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8343.469170] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8343.491799] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8344.444045] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8352.970586] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8357.859475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8357.883905] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8393.698263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8393.704959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.720418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8393.741206] Lustre: Skipped 1 previous similar message [ 8398.830202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8398.846113] Lustre: Skipped 2 previous similar messages [ 8399.874995] Lustre: server umount lustre-OST0000 complete [ 8405.984887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8405.999243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8406.018960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8410.914745] Lustre: server umount lustre-MDT0000 complete [ 8415.114888] LustreError: 255318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085105 with bad export cookie 14361246668086166763 [ 8415.120819] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8415.133402] LustreError: 255318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8415.427356] Lustre: server umount lustre-MDT0001 complete [ 8431.680905] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8435.144736] Key type lgssc unregistered [ 8435.570551] LNet: 260754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8435.576649] LNetError: 260754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8435.604803] LNet: Removed LNI 192.168.204.101@tcp [ 8436.549128] Key type .llcrypt unregistered [ 8436.551741] Key type ._llcrypt unregistered [ 8467.033157] Key type ._llcrypt registered [ 8467.037320] Key type .llcrypt registered [ 8467.174646] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8468.347174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8468.392483] alg: No test for adler32 (adler32-zlib) [ 8469.517589] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8469.804680] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 8471.479249] Key type lgssc registered [ 8472.926123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8482.762220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8482.795020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.515941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8490.454569] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8494.061867] LustreError: 262174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.094816] LustreError: 262174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8499.177053] LustreError: 262173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.808453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8501.259157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8506.839774] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8515.866939] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8523.882758] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8533.031239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8533.606435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.919597] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8549.856752] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8554.183185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8582.128943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8582.135979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8582.146580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8585.187612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8585.189117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8585.213863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8586.655391] Lustre: server umount lustre-OST0000 complete [ 8590.305659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.324804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8590.328748] Lustre: Skipped 2 previous similar messages [ 8595.426736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8595.432744] Lustre: Skipped 1 previous similar message [ 8596.743415] Lustre: server umount lustre-MDT0000 complete [ 8600.547097] LustreError: 264614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.567198] LustreError: 264614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8600.744626] LustreError: 262153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085291 with bad export cookie 12517105920520323962 [ 8600.748857] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8600.754929] LustreError: 262153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8601.177489] Lustre: server umount lustre-MDT0001 complete [ 8613.496732] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8616.565519] Key type lgssc unregistered [ 8616.872923] LNet: 266496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8616.884247] LNetError: 266496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8616.909152] LNet: Removed LNI 192.168.204.101@tcp [ 8617.965171] Key type .llcrypt unregistered [ 8617.975819] Key type ._llcrypt unregistered [ 8639.090319] Key type ._llcrypt registered [ 8639.092882] Key type .llcrypt registered [ 8639.198211] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8640.514252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8640.524205] alg: No test for adler32 (adler32-zlib) [ 8641.726922] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8642.004878] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 8643.704266] Key type lgssc registered [ 8644.878491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8653.830843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8653.864115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.421762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.892091] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8665.569124] LustreError: 267719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8665.590681] LustreError: 267719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8669.770735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.202812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8674.570826] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8682.043359] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8687.776465] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8695.430028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8695.831794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8699.956505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8702.338318] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8709.897512] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8725.475314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8725.496978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8725.522206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8725.990918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8731.105458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8731.114897] Lustre: Skipped 2 previous similar messages [ 8731.558499] Lustre: server umount lustre-OST0000 complete [ 8736.228748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8736.238249] Lustre: Skipped 1 previous similar message [ 8736.247379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8736.257071] Lustre: Skipped 1 previous similar message [ 8741.345220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8741.349554] Lustre: Skipped 1 previous similar message [ 8741.751321] Lustre: server umount lustre-MDT0000 complete [ 8745.673751] LustreError: 267698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085435 with bad export cookie 17580150099427941852 [ 8745.676599] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8745.691915] LustreError: 267698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8745.988761] Lustre: server umount lustre-MDT0001 complete [ 8757.410471] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8760.288752] Key type lgssc unregistered [ 8760.566737] LNet: 271398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8760.576921] LNetError: 271398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8760.592493] LNet: Removed LNI 192.168.204.101@tcp [ 8761.498640] Key type .llcrypt unregistered [ 8761.505084] Key type ._llcrypt unregistered [ 8786.025000] Key type ._llcrypt registered [ 8786.028211] Key type .llcrypt registered [ 8786.165318] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8787.234922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8787.260900] alg: No test for adler32 (adler32-zlib) [ 8788.420729] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8788.673314] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 8790.344621] Key type lgssc registered [ 8791.515990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8799.609456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8799.624936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8801.148378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8806.229375] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8811.491235] LustreError: 272817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8811.515423] LustreError: 272817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8816.320387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.611942] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8816.810184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8821.489388] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8829.398909] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8836.266371] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8843.820870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8844.353499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.470751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8851.282120] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8859.701922] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8891.873767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8891.890234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8891.910383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8891.915209] Lustre: Skipped 1 previous similar message [ 8894.127552] Lustre: server umount lustre-OST0000 complete [ 8898.532533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8898.536854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8898.545566] Lustre: Skipped 1 previous similar message [ 8898.550139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8900.068396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8902.113613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8902.139637] Lustre: Skipped 3 previous similar messages [ 8907.237403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8907.246447] Lustre: Skipped 1 previous similar message [ 8912.351154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8912.601922] Lustre: server umount lustre-MDT0000 complete [ 8916.572055] LustreError: 272797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085606 with bad export cookie 2666360662200643348 [ 8916.578490] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8916.583894] LustreError: 272797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8916.955627] Lustre: server umount lustre-MDT0001 complete [ 8929.117466] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8932.515463] Key type lgssc unregistered [ 8932.883773] LNet: 277125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8932.895426] LNetError: 277125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8932.930917] LNet: Removed LNI 192.168.204.101@tcp [ 8934.254378] Key type .llcrypt unregistered [ 8934.259711] Key type ._llcrypt unregistered [ 8953.654990] Key type ._llcrypt registered [ 8953.657965] Key type .llcrypt registered [ 8953.824348] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8955.546880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8955.561501] alg: No test for adler32 (adler32-zlib) [ 8957.045214] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8957.384546] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 8959.191207] Key type lgssc registered [ 8961.051484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8970.126248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8970.153533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8971.655753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8976.246632] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8980.968440] LustreError: 278363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8981.002304] LustreError: 278363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8986.099308] LustreError: 278365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8987.036829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.414376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8991.665872] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8999.799530] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9005.552470] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9014.108263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9014.674657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.699625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9022.209662] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9031.234701] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9047.531878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9047.533641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9047.552559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9047.564514] Lustre: Skipped 1 previous similar message [ 9047.578127] LustreError: Skipped 1 previous similar message [ 9049.268358] Lustre: server umount lustre-OST0000 complete [ 9057.765368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.783314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.787743] Lustre: Skipped 1 previous similar message [ 9060.665371] Lustre: server umount lustre-MDT0000 complete [ 9062.882120] LustreError: 278367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9062.894236] LustreError: 278367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9065.359969] LustreError: 278330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085755 with bad export cookie 15827175558897911983 [ 9065.361219] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9065.368266] LustreError: 278330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9065.899498] Lustre: server umount lustre-MDT0001 complete [ 9078.333466] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 9081.856675] Key type lgssc unregistered [ 9082.240900] LNet: 282067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9082.245986] LNetError: 282067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9082.272064] LNet: Removed LNI 192.168.204.101@tcp [ 9083.284292] Key type .llcrypt unregistered [ 9083.288616] Key type ._llcrypt unregistered [ 9097.062810] Key type ._llcrypt registered [ 9097.069252] Key type .llcrypt registered [ 9097.223260] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9118.690276] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 9134.066361] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9135.356459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9135.539482] alg: No test for adler32 (adler32-zlib) [ 9136.731688] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 9137.020680] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9138.744290] Key type lgssc registered [ 9140.008607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9176.845995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9176.909737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.175436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9178.243299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9178.329219] Lustre: lustre-MDT0000: new disk, initializing [ 9178.486559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9178.517709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9183.678138] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9195.219262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9195.329684] Lustre: 285622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9195.386291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9195.392883] Lustre: Skipped 1 previous similar message [ 9195.500939] Lustre: lustre-MDT0001: new disk, initializing [ 9195.596304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9195.632874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9195.648292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9200.088047] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9209.537747] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9215.889208] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9223.068034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9223.344198] Lustre: lustre-OST0000: new disk, initializing [ 9223.347112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9223.353884] Lustre: 287593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9223.442213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9229.387181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9229.395951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9229.509686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9230.238208] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9239.804891] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9245.413398] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9245.758550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9252.243937] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9252.644663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9260.514652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9260.519265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9260.520725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9260.531080] LustreError: Skipped 1 previous similar message [ 9260.583914] Lustre: Skipped 1 previous similar message [ 9265.652454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9265.660846] Lustre: Skipped 2 previous similar messages [ 9270.239190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9270.520572] Lustre: server umount lustre-OST0000 complete [ 9275.875644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9275.879252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9275.896753] Lustre: Skipped 1 previous similar message [ 9280.992511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9281.005479] Lustre: Skipped 2 previous similar messages [ 9286.119148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9286.123330] Lustre: Skipped 1 previous similar message [ 9289.186016] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9289.758622] Lustre: server umount lustre-MDT0000 complete [ 9294.865247] LustreError: 286420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085985 with bad export cookie 10324480447608871493 [ 9294.876248] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9294.884318] LustreError: 286420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9295.424326] Lustre: server umount lustre-MDT0001 complete [ 9304.659938] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:13:14 (1785085994) [ 9310.872236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9311.309722] LustreError: 290007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9311.343731] LustreError: 290007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9311.418556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9315.986449] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9316.837939] LustreError: 290008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9320.934970] LustreError: 290007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9325.084715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9325.468836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9330.042559] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9338.244636] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9344.653843] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9352.328502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9352.752866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9360.198157] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9368.196726] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9385.445053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9385.446745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9385.450229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9385.480445] Lustre: Skipped 1 previous similar message [ 9388.346891] Lustre: server umount lustre-OST0000 complete [ 9395.682174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9395.697397] Lustre: Skipped 1 previous similar message [ 9398.832271] Lustre: server umount lustre-MDT0000 complete [ 9400.813200] LustreError: 291239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9400.841686] LustreError: 291239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9403.142873] LustreError: 290742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086093 with bad export cookie 10324480447608872382 [ 9403.148105] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9403.160498] LustreError: 290742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9403.617190] Lustre: server umount lustre-MDT0001 complete [ 9416.060965] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 9419.149932] Key type lgssc unregistered [ 9419.600190] LNet: 293562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9419.606152] LNetError: 293562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9419.625608] LNet: Removed LNI 192.168.204.101@tcp [ 9420.661980] Key type .llcrypt unregistered [ 9420.665713] Key type ._llcrypt unregistered [ 9450.890610] Key type ._llcrypt registered [ 9450.895647] Key type .llcrypt registered [ 9451.063258] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 9468.109053] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9470.301021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9470.373949] alg: No test for adler32 (adler32-zlib) [ 9471.570637] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 9471.973691] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9473.719231] Key type lgssc registered [ 9475.675983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9526.577079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9526.614444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.805922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9527.845671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9527.937526] Lustre: lustre-MDT0000: new disk, initializing [ 9528.052475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9528.077808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9532.905957] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9546.375723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9546.510046] Lustre: 297393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9546.578620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9546.593365] Lustre: Skipped 1 previous similar message [ 9546.687891] Lustre: lustre-MDT0001: new disk, initializing [ 9546.829598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9546.890137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9546.919851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9551.665919] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9562.691084] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9569.261656] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9577.050823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9577.395371] Lustre: lustre-OST2710: new disk, initializing [ 9577.398745] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9577.404567] Lustre: 299365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9577.490226] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9584.255026] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9587.284973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9587.309654] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9587.477203] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9596.817543] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9604.384807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9604.627503] Lustre: lustre-OST03e8: new disk, initializing [ 9604.636881] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9604.646550] Lustre: 300429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9604.776780] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9611.210186] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9612.420139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9612.437811] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9612.534292] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9620.762552] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9628.772512] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9629.050434] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9635.437994] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9635.597934] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9641.187634] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9641.596780] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9647.448157] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9647.884583] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9653.737540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9653.744897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9653.757596] Lustre: Skipped 3 previous similar messages [ 9653.777876] Lustre: Skipped 3 previous similar messages [ 9655.009099] Lustre: server umount lustre-MDT0000 complete [ 9659.406570] LustreError: 299363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086349 with bad export cookie 14471749749589671905 [ 9659.430969] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9659.864928] Lustre: server umount lustre-MDT0001 complete [ 9674.638362] Lustre: server umount lustre-OST2710 complete [ 9689.820569] Lustre: server umount lustre-OST03e8 complete [ 9710.991761] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 9723.067337] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9772.666730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.043434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9773.087798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9773.178874] Lustre: lustre-MDT0000: new disk, initializing [ 9773.254600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9773.270054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9778.474520] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9790.029749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9790.137543] Lustre: 305681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9790.192459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9790.195878] Lustre: Skipped 1 previous similar message [ 9790.291701] Lustre: lustre-MDT0001: new disk, initializing [ 9790.388336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9790.447196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9790.467673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9795.669603] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9807.461818] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9815.935517] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9825.357638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9825.619748] Lustre: lustre-OST0000: new disk, initializing [ 9825.631093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9825.644036] Lustre: 307652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9825.799740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9833.319707] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9835.598224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9835.625882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9835.810759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9845.797631] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9851.441305] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9851.595408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9857.455956] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9857.726499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9861.604289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9861.605964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9861.615293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9861.625918] Lustre: Skipped 1 previous similar message [ 9861.674365] Lustre: Skipped 1 previous similar message [ 9866.720793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9866.741191] Lustre: Skipped 1 previous similar message [ 9871.844274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9874.399132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9874.525504] Lustre: server umount lustre-OST0000 complete [ 9882.082667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9882.109131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9882.111625] Lustre: Skipped 1 previous similar message [ 9884.763681] Lustre: server umount lustre-MDT0000 complete [ 9887.207176] LustreError: 306907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.226025] LustreError: 306907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9888.533095] LustreError: 306487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086578 with bad export cookie 14471749749589673578 [ 9888.541664] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9888.542603] LustreError: 306487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9889.052800] Lustre: server umount lustre-MDT0001 complete [ 9898.180098] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:23:07 (1785086587) [ 9899.919952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9902.541434] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:23:11 (1785086591) [ 9907.394745] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9941.267399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9941.655946] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9941.715649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9941.996168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9948.700526] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9959.132664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.230397] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9959.278515] Lustre: 311514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9965.390558] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9988.415693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9988.770241] Lustre: Denying initial registration attempt from nid 192.168.204.101@tcp, specified as failover [ 9988.782276] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9988.795513] LustreError: 312967:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9988.807489] LustreError: 312967:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9988.820710] LustreError: 312967:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9988.906205] Lustre: server umount lustre-OST0000 complete [ 9988.910157] LustreError: 312967:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9997.797614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9997.801583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9997.817623] Lustre: Skipped 2 previous similar messages [ 9997.828335] Lustre: Skipped 1 previous similar message [10002.916599] LustreError: 311526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.934370] LustreError: 311526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10002.997768] Lustre: server umount lustre-MDT0000 complete [10006.991526] LustreError: 312266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086697 with bad export cookie 14471749749589674467 [10006.995649] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10007.006670] LustreError: 312266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10007.723655] Lustre: server umount lustre-MDT0001 complete [10019.733811] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [10022.544470] Key type lgssc unregistered [10022.839778] LNet: 314090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10022.849287] LNetError: 314090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10022.877386] LNet: Removed LNI 192.168.204.101@tcp [10023.695723] Key type .llcrypt unregistered [10023.697973] Key type ._llcrypt unregistered [10036.692114] Key type ._llcrypt registered [10036.698750] Key type .llcrypt registered [10036.873139] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [10038.091718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10038.197411] alg: No test for adler32 (adler32-zlib) [10039.343933] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [10039.594199] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [10041.327260] Key type lgssc registered [10042.610681] Lustre: Echo OBD driver; http://www.lustre.org/ [10094.215015] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [10105.685704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10105.712371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.051835] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10107.078801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10107.234481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10112.430970] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10122.939977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.141141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10123.195050] Lustre: 316856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10123.457477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10128.483411] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10136.413849] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10142.544589] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10163.377882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10163.515588] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10163.671736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10170.309702] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10180.044345] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10186.731742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10186.741614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.755915] LustreError: Skipped 1 previous similar message [10186.763112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10186.773874] Lustre: Skipped 1 previous similar message [10186.802983] Lustre: Skipped 1 previous similar message [10188.593786] Lustre: server umount lustre-OST0000 complete [10195.424304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10195.434829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10195.457855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10196.962061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10196.972524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10198.822565] Lustre: server umount lustre-MDT0000 complete [10202.446740] LustreError: 316848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086892 with bad export cookie 1612798846367812335 [10202.448800] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10202.453601] LustreError: 316848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10202.836695] Lustre: server umount lustre-MDT0001 complete [10215.620860] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [10218.916426] Key type lgssc unregistered [10219.256085] LNet: 320436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10219.271363] LNetError: 320436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10219.307974] LNet: Removed LNI 192.168.204.101@tcp [10220.251980] Key type .llcrypt unregistered [10220.254198] Key type ._llcrypt unregistered [10247.055916] Key type ._llcrypt registered [10247.060148] Key type .llcrypt registered [10247.208889] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [10248.342462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10248.509356] alg: No test for adler32 (adler32-zlib) [10249.702591] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [10250.001589] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [10251.663233] Key type lgssc registered [10252.840355] Lustre: Echo OBD driver; http://www.lustre.org/ [10261.840933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10261.864280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10263.366119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10267.680503] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10273.766653] LustreError: 321856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.800843] LustreError: 321856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10277.222090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10277.539232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10281.017961] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10289.922495] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10296.026477] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10305.666328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10306.076699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10313.016167] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10321.965721] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10332.645594] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10332.658605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10332.660913] Lustre: Skipped 1 previous similar message [10332.680816] Lustre: Skipped 2 previous similar messages [10335.249326] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.1@tcp (stopping) [10337.762239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10337.771594] Lustre: Skipped 2 previous similar messages [10338.847680] Lustre: server umount lustre-MDT0000 complete [10340.331224] LustreError: 324434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.882404] LustreError: 321855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.916902] LustreError: 321855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10343.645588] LustreError: 321836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087033 with bad export cookie 9988132961786048084 [10343.658053] LustreError: 321836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10343.660682] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10345.459535] LustreError: 323696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.495821] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.1@tcp (stopping) [10348.001220] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.028705] Lustre: Skipped 1 previous similar message [10350.582336] Lustre: server umount lustre-MDT0001 complete [10358.486327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10368.111784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10368.480197] LustreError: 321164:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b234bf90 x1871799323230720/t0(0) o250->MGC192.168.204.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10368.809490] LustreError: 323686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10368.828584] LustreError: 323686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10368.881047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10368.969163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10374.535595] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10378.101320] LustreError: 323687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.109659] LustreError: 323687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10378.113345] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10386.152131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10386.597770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10392.048380] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10392.399146] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10403.648704] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10405.715910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10413.755794] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10416.139743] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10425.314751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10425.329403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10425.354634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10425.363670] Lustre: Skipped 2 previous similar messages [10427.881990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10427.882977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10429.337227] Lustre: server umount lustre-OST0000 complete [10437.601432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10437.606407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10437.623479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10437.630178] Lustre: Skipped 2 previous similar messages [10439.762458] Lustre: server umount lustre-MDT0000 complete [10443.248315] LustreError: 325449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10443.289947] LustreError: 325449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10444.219322] LustreError: 321836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087134 with bad export cookie 9988132961786055224 [10444.224532] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10444.232697] LustreError: 321836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10444.789285] Lustre: server umount lustre-MDT0001 complete [10457.354593] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [10460.517356] Key type lgssc unregistered [10460.849520] LNet: 328120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10460.862643] LNetError: 328120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10460.877788] LNet: Removed LNI 192.168.204.101@tcp [10461.976047] Key type .llcrypt unregistered [10461.980996] Key type ._llcrypt unregistered [10491.191610] Key type ._llcrypt registered [10491.197669] Key type .llcrypt registered [10491.340915] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [10492.331104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10492.402295] alg: No test for adler32 (adler32-zlib) [10493.600296] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [10493.816395] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [10495.495345] Key type lgssc registered [10496.554886] Lustre: Echo OBD driver; http://www.lustre.org/ [10504.457318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10504.481109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.956983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10511.493321] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10516.453382] LustreError: 329539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10516.483883] LustreError: 329539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10521.257774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10521.734412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10526.403620] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10534.845668] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10541.089255] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10544.859340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10544.867230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10544.874832] Lustre: Skipped 1 previous similar message [10544.892331] Lustre: Skipped 1 previous similar message [10545.957918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10545.974141] Lustre: Skipped 1 previous similar message [10551.207829] Lustre: server umount lustre-MDT0000 complete [10551.274376] LustreError: 329536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.290348] LustreError: 329536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10555.020185] LustreError: 329521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087245 with bad export cookie 10050041231750251029 [10555.031601] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10555.423830] Lustre: server umount lustre-MDT0001 complete [10571.674705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10572.115626] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10572.178947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10572.390281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10577.553211] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10587.432071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10587.545417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10587.594903] Lustre: 332126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10587.966461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10592.819975] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10601.119780] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10608.067534] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10619.122509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10619.451729] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10619.466579] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10619.480699] LustreError: 334008:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10619.487418] LustreError: 334008:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10619.497056] LustreError: 334008:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10619.633804] Lustre: server umount lustre-OST0000 complete [10619.640722] LustreError: 334008:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10631.533554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10631.850974] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10632.049935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10638.765417] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10641.739260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10649.225683] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10658.928551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10659.060219] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10659.207626] Lustre: lustre-OST0001: new disk, initializing [10659.215602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10659.241269] Lustre: 335552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10659.384711] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10666.184700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10666.209978] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10666.356961] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10668.134674] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10677.297215] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10681.826440] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10681.840409] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.881127] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10681.891311] Lustre: Skipped 1 previous similar message [10686.849876] Lustre: server umount lustre-OST0001 complete [10686.950777] LustreError: 334474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10686.975700] LustreError: 334474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10692.071397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10692.095355] LustreError: 335831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10692.095460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10692.117683] Lustre: Skipped 2 previous similar messages [10692.149340] LustreError: 335831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10697.185417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10697.186030] LustreError: 335831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10697.206670] Lustre: Skipped 2 previous similar messages [10697.245295] LustreError: 335831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10697.721424] Lustre: server umount lustre-OST0000 complete [10702.309312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10707.426831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10707.446246] Lustre: Skipped 3 previous similar messages [10708.479221] Lustre: server umount lustre-MDT0000 complete [10712.541081] LustreError: 332117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087402 with bad export cookie 10050041231750251582 [10712.554921] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10712.563173] LustreError: 332117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10712.970679] Lustre: server umount lustre-MDT0001 complete [10724.962585] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [10727.774346] Key type lgssc unregistered [10728.094355] LNet: 337512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10728.104971] LNetError: 337512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10728.119300] LNet: Removed LNI 192.168.204.101@tcp [10729.205994] Key type .llcrypt unregistered [10729.208813] Key type ._llcrypt unregistered [10812.296455] Key type ._llcrypt registered [10812.301483] Key type .llcrypt registered [10812.476312] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [10833.235737] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [10834.835703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10834.965554] alg: No test for adler32 (adler32-zlib) [10836.072341] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [10836.438325] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [10838.239223] Key type lgssc registered [10839.576943] Lustre: Echo OBD driver; http://www.lustre.org/ [10884.766247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10884.807487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10886.150984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10886.234061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10886.362928] Lustre: lustre-MDT0000: new disk, initializing [10886.504922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10886.545088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10892.864318] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10908.057436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10908.271191] Lustre: 342451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10908.378315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10908.403204] Lustre: Skipped 1 previous similar message [10908.593099] Lustre: lustre-MDT0001: new disk, initializing [10908.701896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10908.759506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10908.767753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10916.371327] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10928.689382] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10936.646427] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10945.075696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.367365] Lustre: lustre-OST0000: new disk, initializing [10945.373963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10945.386087] Lustre: 344422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10945.514500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10951.782741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10951.791922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10951.934237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10954.205682] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10967.077754] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10974.005270] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10974.267820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10980.510874] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10980.801819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10988.002648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10988.007292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.014920] LustreError: Skipped 1 previous similar message [10988.019669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10988.031220] Lustre: Skipped 1 previous similar message [10989.913516] Lustre: server umount lustre-OST0000 complete [10995.684724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10995.690694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10995.700692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10995.705188] Lustre: Skipped 1 previous similar message [10998.241500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10998.260400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10998.273937] Lustre: Skipped 1 previous similar message [11000.482368] Lustre: server umount lustre-MDT0000 complete [11004.997440] LustreError: 342444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087695 with bad export cookie 858809683003942539 [11005.004161] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11005.008041] LustreError: 342444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11005.534893] Lustre: server umount lustre-MDT0001 complete [11015.503508] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:41:45 (1785087705) [11025.067974] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:41:54 (1785087714) [11036.861266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11038.146158] LustreError: 347181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11038.172447] LustreError: 347181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11038.271316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11043.301781] LustreError: 347182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11043.834732] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11048.425826] LustreError: 347181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11053.543491] LustreError: 347182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11053.760857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11054.131343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11059.410780] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11068.474809] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11074.806387] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11081.669371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11082.020109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11088.916296] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11097.261902] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11106.215132] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11108.045897] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11109.838896] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11111.443873] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11113.341652] Lustre: Failing over lustre-MDT0000 [11113.610366] Lustre: server umount lustre-MDT0000 complete [11114.986091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11114.996100] LustreError: 347178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11115.001162] Lustre: Skipped 1 previous similar message [11115.044692] LustreError: 347178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11125.217869] LustreError: 347182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.241260] LustreError: 347182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11131.359482] Lustre: 340678:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785087805/real 1785087805] req@00000000ec29cc14 x1871799937740672/t0(0) o400->MGC192.168.204.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785087821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11131.399828] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11132.613600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11141.602724] LustreError: 340674:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dc2b02a8 x1871799937745792/t0(0) o250->MGC192.168.204.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11141.632755] LustreError: 347909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.651425] LustreError: 347909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [11141.859319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11141.920687] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11142.646051] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11146.256566] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11147.262910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11147.303109] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11147.366603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11157.106775] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11158.593387] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11162.239942] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11164.635545] Lustre: Failing over lustre-MDT0000 [11165.038794] Lustre: server umount lustre-MDT0000 complete [11167.720473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11167.722124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11167.746305] Lustre: Skipped 3 previous similar messages [11177.959829] LustreError: 347181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11178.001313] LustreError: 347181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11183.570061] Lustre: 340678:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785087857/real 1785087857] req@00000000c166882d x1871799937761536/t0(0) o400->MGC192.168.204.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785087873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11183.593877] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11185.030498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.800067] LustreError: 340674:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000013c5839e x1871799937766528/t0(0) o250->MGC192.168.204.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11193.097195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11193.171728] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11193.842967] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11198.506065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11198.514583] Lustre: Skipped 2 previous similar messages [11198.536344] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11198.609240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11200.039524] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11212.952530] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11215.783700] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11224.035063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.050306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11224.064435] Lustre: Skipped 2 previous similar messages [11224.089728] Lustre: Skipped 2 previous similar messages [11227.804759] Lustre: server umount lustre-MDT0000 complete [11231.887464] LustreError: 347163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785087922 with bad export cookie 858809683003945248 [11231.891825] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11231.905856] LustreError: 347163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11232.337594] Lustre: server umount lustre-MDT0001 complete [11246.354493] Lustre: server umount lustre-OST0000 complete [11267.434825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11267.900368] LustreError: 353526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11267.914340] LustreError: 353526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [11267.982731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11273.120636] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11283.109235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.912453] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11296.261377] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11302.559478] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11310.735619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11313.141655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11318.253474] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11326.978459] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11338.726906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11338.727691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11338.750280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11338.763902] Lustre: Skipped 1 previous similar message [11338.785818] Lustre: Skipped 1 previous similar message [11344.523654] Lustre: server umount lustre-OST0000 complete [11348.971954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11348.981931] Lustre: Skipped 3 previous similar messages [11363.295566] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11363.644478] Lustre: server umount lustre-MDT0000 complete [11368.364458] LustreError: 353508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088058 with bad export cookie 858809683003945997 [11368.369577] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11368.373242] LustreError: 353508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11368.764918] Lustre: server umount lustre-MDT0001 complete [11382.175428] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [11384.950830] Key type lgssc unregistered [11385.251467] LNet: 357060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11385.269467] LNetError: 357060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11385.299708] LNet: Removed LNI 192.168.204.101@tcp [11386.162618] Key type .llcrypt unregistered [11386.167614] Key type ._llcrypt unregistered [11415.152293] Key type ._llcrypt registered [11415.155648] Key type .llcrypt registered [11415.275622] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [11416.397680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11416.454941] alg: No test for adler32 (adler32-zlib) [11417.677871] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [11418.018041] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [11419.687171] Key type lgssc registered [11420.928815] Lustre: Echo OBD driver; http://www.lustre.org/ [11430.096878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11430.118855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.678941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11436.722661] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11441.133418] LustreError: 358579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11441.178551] LustreError: 358579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11446.240253] LustreError: 358580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.660365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11450.224089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11455.641807] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11464.669900] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11472.237567] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11479.988731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11480.375878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11487.998718] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11497.019825] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11500.872039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11516.387834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11516.391251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11516.402041] Lustre: Skipped 2 previous similar messages [11516.407964] Lustre: Skipped 1 previous similar message [11520.545430] Lustre: server umount lustre-MDT0000 complete [11521.506176] LustreError: 358574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11521.528824] LustreError: 358574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11524.781632] LustreError: 358560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088215 with bad export cookie 12417157616099715582 [11524.788085] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11524.794157] LustreError: 358560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11525.140610] Lustre: server umount lustre-MDT0001 complete [11538.997074] Lustre: server umount lustre-OST0000 complete [11553.524339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.121224] LustreError: 362315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11554.231196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11558.644978] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11559.395299] LustreError: 362316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.674576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11568.184817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11572.823815] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11580.514786] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11586.318337] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11595.162210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11595.526636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11599.616103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11603.333315] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11612.027052] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11628.525943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11628.532821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11628.545342] Lustre: Skipped 2 previous similar messages [11628.555833] Lustre: Skipped 2 previous similar messages [11631.041822] Lustre: server umount lustre-MDT0000 complete [11633.646156] LustreError: 362315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.682829] LustreError: 362315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11635.141735] LustreError: 362297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088325 with bad export cookie 12417157616099716702 [11635.148445] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11635.155876] LustreError: 362297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11635.656552] Lustre: server umount lustre-MDT0001 complete [11650.157507] Lustre: server umount lustre-OST0000 complete [11667.042736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.673976] LustreError: 366224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.782847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11673.531873] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11685.143323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11693.965923] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11703.239787] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11709.671130] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11716.063584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11716.497134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11716.503209] Lustre: Skipped 1 previous similar message [11722.641275] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11730.448900] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11737.015606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11747.301751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11747.304185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11747.311790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11747.311800] Lustre: Skipped 1 previous similar message [11747.338259] Lustre: Skipped 1 previous similar message [11751.828974] Lustre: server umount lustre-OST0000 complete [11757.536436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11757.542197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11757.560877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11757.560884] Lustre: Skipped 1 previous similar message [11757.580378] Lustre: Skipped 1 previous similar message [11761.869046] Lustre: server umount lustre-MDT0000 complete [11762.659994] LustreError: 367582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.699456] LustreError: 367582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11765.530088] LustreError: 366204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088455 with bad export cookie 12417157616099717724 [11765.533483] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11765.547517] LustreError: 366204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11765.926787] Lustre: server umount lustre-MDT0001 complete [11777.847275] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [11780.927078] Key type lgssc unregistered [11781.355857] LNet: 369783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11781.384793] LNetError: 369783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11781.421897] LNet: Removed LNI 192.168.204.101@tcp [11782.428077] Key type .llcrypt unregistered [11782.431749] Key type ._llcrypt unregistered [11810.026988] Key type ._llcrypt registered [11810.034613] Key type .llcrypt registered [11810.229436] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [11811.967600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11812.021485] alg: No test for adler32 (adler32-zlib) [11813.188516] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [11813.417263] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [11815.079224] Key type lgssc registered [11816.377594] Lustre: Echo OBD driver; http://www.lustre.org/ [11824.239993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11824.267284] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11824.283777] LustreError: 371227:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11824.312758] LustreError: 371227:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11824.320582] LustreError: 371227:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11824.328224] LustreError: 371227:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11824.335930] LustreError: 371227:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11833.280612] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11833.294075] LustreError: 371626:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11833.332887] LustreError: 371626:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11833.341251] LustreError: 371626:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11833.348939] LustreError: 371626:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11833.361888] LustreError: 371626:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11853.400153] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [11856.561318] Key type lgssc unregistered [11856.855760] LNet: 372629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11856.869398] LNetError: 372629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11856.894567] LNet: Removed LNI 192.168.204.101@tcp [11857.839602] Key type .llcrypt unregistered [11857.849128] Key type ._llcrypt unregistered [11874.815124] Key type ._llcrypt registered [11874.819966] Key type .llcrypt registered [11874.905467] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [11890.280851] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [11891.602568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11891.652536] alg: No test for adler32 (adler32-zlib) [11893.014545] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [11893.389946] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [11895.151182] Key type lgssc registered [11896.894888] Lustre: Echo OBD driver; http://www.lustre.org/ [11941.650231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11941.703063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11943.069518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11943.117794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11943.300214] Lustre: lustre-MDT0000: new disk, initializing [11943.469572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11943.504117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11949.181322] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11961.396254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11961.499809] Lustre: 375786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11961.566711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11961.571188] Lustre: Skipped 1 previous similar message [11961.638195] Lustre: lustre-MDT0001: new disk, initializing [11961.730856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11961.786027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11961.800567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11966.336168] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11976.996366] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11983.107053] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11990.487925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11990.798784] Lustre: lustre-OST0000: new disk, initializing [11990.802785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11990.810202] Lustre: 377753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11990.977770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11996.715060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11996.722400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11996.820325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11997.037805] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12006.618378] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12011.420351] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12011.704482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12016.189743] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12016.417717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12022.756747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12022.758489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12022.774939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12022.792451] LustreError: Skipped 1 previous similar message [12024.459202] Lustre: server umount lustre-OST0000 complete [12032.997122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12033.005357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12033.015188] Lustre: Skipped 2 previous similar messages [12033.027907] Lustre: Skipped 2 previous similar messages [12035.529470] Lustre: server umount lustre-MDT0000 complete [12040.182334] LustreError: 375779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088730 with bad export cookie 16146781403133834274 [12040.185590] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12040.196962] LustreError: 375779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12040.744743] Lustre: server umount lustre-MDT0001 complete [12048.390116] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12050.102446] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:58:59 (1785088739) [12056.018247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12056.613067] LustreError: 380215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12056.645781] LustreError: 380215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12056.700332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12060.638110] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12062.178251] LustreError: 380216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12066.277184] LustreError: 380215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.693081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12069.002667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12073.181796] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12080.763108] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12087.043879] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12095.041298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12095.502969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12103.843095] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12114.490208] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12123.747322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12124.011382] Lustre: lustre-OST0001: new disk, initializing [12124.017065] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12124.025253] Lustre: 383056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12124.172761] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12131.513950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12131.539236] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12131.745757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12132.552447] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12144.553110] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12151.807697] Lustre: lustre-OST0001: Not available for connect from 192.168.204.1@tcp (stopping) [12152.297512] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12152.297571] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12155.083840] Lustre: server umount lustre-OST0001 complete [12156.912449] LustreError: 382834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12162.538191] LustreError: 382039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12162.555415] LustreError: 382039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12172.781574] LustreError: 383983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12172.812181] LustreError: 383983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12173.297428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12173.308958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12173.336548] Lustre: Skipped 1 previous similar message [12173.360827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12173.371295] Lustre: Skipped 2 previous similar messages [12175.337997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12177.897026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12177.914174] Lustre: Skipped 2 previous similar messages [12179.204402] Lustre: server umount lustre-OST0000 complete [12186.594659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12186.608145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12186.635438] Lustre: Skipped 1 previous similar message [12186.645859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12190.675771] Lustre: server umount lustre-MDT0000 complete [12193.251455] LustreError: 380215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12193.271139] LustreError: 380215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12196.131194] LustreError: 380197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785088886 with bad export cookie 16146781403133835156 [12196.134738] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12196.148455] LustreError: 380197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12202.580891] Lustre: server umount lustre-MDT0001 complete [12215.690463] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [12218.437693] Key type lgssc unregistered [12218.761592] LNet: 385056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12218.772682] LNetError: 385056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12218.791832] LNet: Removed LNI 192.168.204.101@tcp [12219.870383] Key type .llcrypt unregistered [12219.873238] Key type ._llcrypt unregistered [12276.426718] Key type ._llcrypt registered [12276.429527] Key type .llcrypt registered [12276.617444] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [12278.269595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12278.338448] alg: No test for adler32 (adler32-zlib) [12279.445189] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [12279.700107] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [12281.463174] Key type lgssc registered [12282.633536] Lustre: Echo OBD driver; http://www.lustre.org/ [12290.811243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12290.830526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.191930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12292.218391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12292.332823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12297.619820] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12307.063066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12307.252872] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12307.290973] Lustre: 387857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12307.551475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12313.820652] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12322.546518] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12329.026638] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12333.024806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12333.031883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12333.043786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12335.586491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12335.589234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12337.808886] Lustre: server umount lustre-MDT0000 complete [12341.646251] LustreError: 387849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089031 with bad export cookie 15497854352058218232 [12341.661718] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12341.664166] LustreError: 387849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12342.217452] Lustre: server umount lustre-MDT0001 complete [12352.985661] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:04:02 (1785089042) [12359.897440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12360.327322] LustreError: 390370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.355602] LustreError: 390370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12360.393152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12364.950515] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12365.803834] LustreError: 390371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12370.919668] LustreError: 390370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12374.007679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12374.302310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12378.870681] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12387.155599] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12393.947962] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12402.699047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12402.908435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12403.194648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12409.862739] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12419.689200] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12427.499146] Lustre: Permanently deactivating lustre-OST0000 [12427.534614] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12436.042427] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12437.821606] LustreError: 393361:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12437.833652] LustreError: 393361:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12437.838512] LustreError: 393361:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12440.973384] LustreError: 393409:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12440.986783] LustreError: 393409:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12449.477478] Lustre: server umount lustre-OST0000 complete [12452.181790] LustreError: 393659:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12452.190570] LustreError: 393659:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12456.416778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12456.424114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12456.440478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12456.449052] Lustre: Skipped 1 previous similar message [12462.566729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12462.572296] Lustre: Skipped 3 previous similar messages [12464.716824] Lustre: server umount lustre-MDT0000 complete [12467.681572] LustreError: 390365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12467.696087] LustreError: 390365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12468.119395] LustreError: 390352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089158 with bad export cookie 15497854352058218813 [12468.125412] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12468.127322] LustreError: 390352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12478.633381] Lustre: server umount lustre-MDT0001 complete [12487.174245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.731451] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12495.555439] LustreError: 394817:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12502.625972] Lustre: 395156:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.101@tcp [12505.256512] Lustre: server umount MGS complete [12517.396412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12518.118465] LustreError: 395883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12518.161080] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12518.169736] Lustre: Skipped 1 previous similar message [12518.245715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12522.069693] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12528.621483] LustreError: 395883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12528.650623] LustreError: 395883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12530.972856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12531.261711] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12536.478315] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12545.214672] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12550.993544] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12552.807462] Lustre: Permanently reactivating lustre-OST0000 [12552.823853] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12552.833097] Lustre: Skipped 2 previous similar messages [12582.373365] LustreError: 397711:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12589.131389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12589.496724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12589.507922] Lustre: Skipped 1 previous similar message [12595.080208] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12602.734162] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12607.972973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12607.974361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12607.983265] LustreError: Skipped 1 previous similar message [12607.993561] Lustre: Skipped 1 previous similar message [12607.998646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12608.012736] Lustre: Skipped 1 previous similar message [12614.158281] Lustre: server umount lustre-OST0000 complete [12618.211609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12618.212571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12618.212840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12618.212845] Lustre: Skipped 2 previous similar messages [12618.235751] Lustre: Skipped 1 previous similar message [12623.545749] Lustre: server umount lustre-MDT0000 complete [12627.507342] LustreError: 395864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089317 with bad export cookie 15497854352058219863 [12627.518404] LustreError: 395864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12627.520209] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12627.929716] Lustre: server umount lustre-MDT0001 complete [12638.341956] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [12641.228505] Key type lgssc unregistered [12641.610140] LNet: 400294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12641.622123] LNetError: 400294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12641.652991] LNet: Removed LNI 192.168.204.101@tcp [12642.691811] Key type .llcrypt unregistered [12642.693862] Key type ._llcrypt unregistered [12664.257788] Key type ._llcrypt registered [12664.262266] Key type .llcrypt registered [12664.381872] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [12681.133662] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [12682.386890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12682.625347] alg: No test for adler32 (adler32-zlib) [12683.703519] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [12683.950864] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [12685.607176] Key type lgssc registered [12686.715939] Lustre: Echo OBD driver; http://www.lustre.org/ [12723.517903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12723.548884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12724.863639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12724.892264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12725.048203] Lustre: lustre-MDT0000: new disk, initializing [12725.143407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12725.176434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12729.972607] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12741.252780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12741.367393] Lustre: 403454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12741.416215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12741.419100] Lustre: Skipped 1 previous similar message [12741.497764] Lustre: lustre-MDT0001: new disk, initializing [12741.584247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12741.648219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12741.676209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12747.163563] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12758.313323] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12764.887606] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12772.714129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12772.949906] Lustre: lustre-OST0000: new disk, initializing [12772.956432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12772.961556] Lustre: 405422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12773.063412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12779.718897] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12780.673437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12780.688385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12780.828108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12790.138128] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12795.567844] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12795.850070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12801.642962] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12801.867048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12806.638726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12806.643323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12806.652858] LustreError: Skipped 1 previous similar message [12806.659762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12806.676874] Lustre: Skipped 1 previous similar message [12806.696709] Lustre: Skipped 1 previous similar message [12810.622398] Lustre: server umount lustre-OST0000 complete [12816.865904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12816.893889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12816.898112] Lustre: Skipped 1 previous similar message [12822.046816] Lustre: server umount lustre-MDT0000 complete [12827.111515] LustreError: 404622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.146369] LustreError: 404622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12827.610319] LustreError: 403447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089517 with bad export cookie 16441147824811800547 [12827.619043] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12827.627517] LustreError: 403447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12828.104231] Lustre: server umount lustre-MDT0001 complete [12841.089683] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12842.900606] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:12:12 (1785089532) [12851.176197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12851.788469] LustreError: 407885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.852481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12856.425325] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12857.316443] LustreError: 407886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12862.436058] LustreError: 407885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12866.588274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12867.000355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12871.674854] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12879.635955] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12886.297235] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12893.424281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12893.807667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12900.598461] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12908.706724] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12911.179881] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12914.209756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12922.850581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12922.854559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12922.856075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12922.865929] LustreError: Skipped 1 previous similar message [12922.906467] Lustre: Skipped 2 previous similar messages [12926.150996] Lustre: server umount lustre-OST0000 complete [12933.089097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12933.096465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12933.115097] Lustre: Skipped 1 previous similar message [12933.122066] Lustre: Skipped 1 previous similar message [12938.221271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12938.231069] Lustre: Skipped 1 previous similar message [12944.354056] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12944.552867] Lustre: server umount lustre-MDT0000 complete [12948.458801] LustreError: 408614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12948.482078] LustreError: 408614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12948.597244] LustreError: 407867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089638 with bad export cookie 16441147824811801429 [12948.610251] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12949.054315] Lustre: server umount lustre-MDT0001 complete [12960.917227] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [12963.868454] Key type lgssc unregistered [12964.231812] LNet: 411611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12964.239423] LNetError: 411611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12964.266493] LNet: Removed LNI 192.168.204.101@tcp [12965.267941] Key type .llcrypt unregistered [12965.270091] Key type ._llcrypt unregistered [12992.430655] Key type ._llcrypt registered [12992.433544] Key type .llcrypt registered [12992.560257] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [12992.861493] Key type .llcrypt unregistered [12992.867877] Key type ._llcrypt unregistered [13012.466070] Key type ._llcrypt registered [13012.469574] Key type .llcrypt registered [13012.645654] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [13013.971854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13014.003274] alg: No test for adler32 (adler32-zlib) [13015.153817] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [13015.435468] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [13017.103155] Key type lgssc registered [13018.680747] Lustre: Echo OBD driver; http://www.lustre.org/ [13027.499894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13027.544197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13029.089824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13034.448877] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13039.585694] LustreError: 413829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13039.612753] LustreError: 413829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13044.705366] LustreError: 413830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13045.417383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13045.762240] LustreError: 413829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13045.787195] LustreError: 413829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13045.807928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13050.853323] LustreError: 413830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13050.878885] LustreError: 413830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13053.260767] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13053.942623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13059.043894] LustreError: 413825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13059.096527] LustreError: 413825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13061.661558] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13068.410473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13068.756452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13072.845129] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13084.134572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13084.154174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13084.158241] Lustre: Skipped 1 previous similar message [13084.161644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13084.199418] Lustre: Skipped 1 previous similar message [13089.056976] Lustre: server umount lustre-OST0000 complete [13094.378853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13094.392378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13094.395365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13094.424130] Lustre: Skipped 1 previous similar message [13099.101511] Lustre: server umount lustre-MDT0000 complete [13099.490539] LustreError: 413830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13099.505875] LustreError: 413830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13103.804853] LustreError: 413810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089794 with bad export cookie 16426130465895791117 [13103.807248] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13103.827872] LustreError: 413810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13104.203045] Lustre: server umount lustre-MDT0001 complete [13116.942278] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [13122.212254] Key type lgssc unregistered [13122.541438] LNet: 416977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13122.552519] LNetError: 416977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13122.570336] LNet: Removed LNI 192.168.204.101@tcp [13123.644743] Key type .llcrypt unregistered [13123.653269] Key type ._llcrypt unregistered [13155.941033] Key type ._llcrypt registered [13155.943270] Key type .llcrypt registered [13156.126852] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [13157.564031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13157.648340] alg: No test for adler32 (adler32-zlib) [13158.755322] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [13159.111533] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [13160.751605] Key type lgssc registered [13162.004228] Lustre: Echo OBD driver; http://www.lustre.org/ [13171.263749] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13171.286636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13186.849276] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13191.967505] LustreError: 418371:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [13218.655472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13226.312746] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13234.862877] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13241.474409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13241.902789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13241.956891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13246.458707] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13246.945261] LustreError: 418397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13246.978905] LustreError: 418397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13252.064473] LustreError: 418396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13253.096254] LustreError: 419411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13253.130631] LustreError: 419411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13255.178295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13255.635277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13260.349298] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13268.726551] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13274.412615] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13281.266925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13281.280636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13281.286736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13281.299481] Lustre: Skipped 1 previous similar message [13282.952784] Lustre: server umount lustre-OST0000 complete [13286.789298] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13286.802406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13286.815257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13286.818939] Lustre: Skipped 1 previous similar message [13289.021994] Lustre: server umount lustre-MDT0000 complete [13291.490390] LustreError: 419415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13291.493314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13293.211429] LustreError: 418379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785089983 with bad export cookie 5979064188680899623 [13293.229644] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13293.590935] Lustre: server umount lustre-MDT0001 complete [13305.807148] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [13309.378480] Key type lgssc unregistered [13309.661641] LNet: 421940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13309.679768] LNetError: 421940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13309.700443] LNet: Removed LNI 192.168.204.101@tcp [13310.803964] Key type .llcrypt unregistered [13310.806808] Key type ._llcrypt unregistered [13341.218260] Key type ._llcrypt registered [13341.221702] Key type .llcrypt registered [13341.320217] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [13342.546319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13342.584553] alg: No test for adler32 (adler32-zlib) [13344.030346] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [13344.406988] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [13346.128757] Key type lgssc registered [13347.565427] Lustre: Echo OBD driver; http://www.lustre.org/ [13354.930618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13354.947133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13356.506956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13363.090292] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13366.758313] LustreError: 423358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13366.775825] LustreError: 423358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13371.879378] LustreError: 423357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13372.718207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13373.199736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13377.628703] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13385.811183] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13391.077745] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13397.647841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13397.936970] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [13398.077735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13402.168139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13404.619624] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13413.034316] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13419.495398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13419.503513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13419.522529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13421.067209] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.1@tcp (stopping) [13422.566353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13422.588494] Lustre: Skipped 1 previous similar message [13423.995756] Lustre: server umount lustre-MDT0000 complete [13426.557758] LustreError: 423339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.1@tcp arrived at 1785090116 with bad export cookie 12124103335361965114 [13426.569878] LustreError: 423339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13427.685651] LustreError: 423357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13427.726506] LustreError: 423357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13428.872907] LustreError: 423338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785090119 with bad export cookie 12124103335361964218 [13428.888780] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13432.800028] LustreError: 423358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13432.803237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13432.804736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13432.816241] LustreError: 423358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13432.841133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13432.845955] Lustre: Skipped 3 previous similar messages [13435.160775] Lustre: server umount lustre-OST0000 complete [13438.946512] LustreError: 423358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13438.963553] LustreError: 423358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13442.459814] Lustre: server umount lustre-MDT0001 complete [13453.711067] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [13456.767564] Key type lgssc unregistered [13457.168489] LNet: 427042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13457.192726] LNetError: 427042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13457.234697] LNet: Removed LNI 192.168.204.101@tcp [13458.436948] Key type .llcrypt unregistered [13458.440855] Key type ._llcrypt unregistered [13485.217112] Key type ._llcrypt registered [13485.219590] Key type .llcrypt registered [13485.322414] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [13486.240383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13486.322353] alg: No test for adler32 (adler32-zlib) [13487.388805] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [13487.622585] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [13489.359271] Key type lgssc registered [13490.711422] Lustre: Echo OBD driver; http://www.lustre.org/ [13498.802996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13498.831705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13500.268212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13505.533914] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13510.626709] LustreError: 428461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13510.639277] LustreError: 428461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13511.663564] LustreError: 428460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13515.695207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13516.036243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13520.981975] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13530.953105] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13538.059545] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13547.574328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13548.005897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13555.221472] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13563.117471] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13571.585744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13577.696965] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13577.713624] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13577.729673] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13579.308367] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.1@tcp (stopping) [13581.797400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13581.800558] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13583.537694] Lustre: server umount lustre-MDT0001 complete [13584.363109] LustreError: 428456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13586.916937] LustreError: 428460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13586.955297] LustreError: 428460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13592.033402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13592.034977] LustreError: 429461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13592.068937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13592.107095] Lustre: Skipped 1 previous similar message [13596.993390] Lustre: server umount lustre-OST0000 complete [13602.104782] Lustre: server umount lustre-MDT0000 complete [13618.732602] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [13622.496719] Key type lgssc unregistered [13622.845739] LNet: 432169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13622.854085] LNetError: 432169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13622.878695] LNet: Removed LNI 192.168.204.101@tcp [13623.858864] Key type .llcrypt unregistered [13623.860653] Key type ._llcrypt unregistered [13652.213473] Key type ._llcrypt registered [13652.222768] Key type .llcrypt registered [13652.365833] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [13674.949206] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [13676.322517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13676.429874] alg: No test for adler32 (adler32-zlib) [13677.715376] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [13677.970303] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [13679.711562] Key type lgssc registered [13681.345284] Lustre: Echo OBD driver; http://www.lustre.org/ [13723.053316] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [13737.982480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13738.004225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13739.367678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13739.461838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13739.535312] Lustre: lustre-MDT0000: new disk, initializing [13739.592310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13739.602522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13745.251928] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13759.779882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13760.024931] Lustre: lustre-OST0000: new disk, initializing [13760.027589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13760.031119] Lustre: Skipped 1 previous similar message [13760.035123] Lustre: 437204:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13760.104817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13766.075438] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13769.837153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13769.849159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13769.875837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13775.815745] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13792.336768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13792.467864] Lustre: 436186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13792.523282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13792.600524] Lustre: lustre-MDT0001: new disk, initializing [13792.739341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13792.794532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13792.805914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13797.582802] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13807.699543] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13807.931465] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13812.383634] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13812.573364] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13817.319568] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13817.325063] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13817.340491] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13817.350398] Lustre: Skipped 1 previous similar message [13817.356637] Lustre: Skipped 1 previous similar message [13818.902572] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.1@tcp (stopping) [13822.448908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13822.464454] Lustre: Skipped 1 previous similar message [13823.715740] Lustre: server umount lustre-MDT0001 complete [13829.103514] LustreError: 436196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13829.123305] LustreError: 436196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13832.678201] LustreError: 436194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13832.699082] LustreError: 436194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13837.792931] LustreError: 436199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13837.821829] LustreError: 436199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13842.912352] LustreError: 436200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13842.917462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13842.960879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13848.038641] LustreError: 436199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13848.043669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13848.471705] Lustre: server umount lustre-OST0000 complete [13858.588776] Lustre: server umount lustre-MDT0000 complete [13874.261423] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [13877.913334] Key type lgssc unregistered [13878.229493] LNet: 440957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13878.246157] LNetError: 440957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13878.274401] LNet: Removed LNI 192.168.204.101@tcp [13879.179809] Key type .llcrypt unregistered [13879.182901] Key type ._llcrypt unregistered [13921.393414] Key type ._llcrypt registered [13921.395068] Key type .llcrypt registered [13921.510542] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [13921.762189] Key type .llcrypt unregistered [13921.764896] Key type ._llcrypt unregistered [13941.484051] Key type ._llcrypt registered [13941.486025] Key type .llcrypt registered [13941.569544] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [13942.795202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13942.851490] alg: No test for adler32 (adler32-zlib) [13944.208923] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [13944.549388] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [13946.255387] Key type lgssc registered [13948.020701] Lustre: Echo OBD driver; http://www.lustre.org/ [13975.717711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13975.742338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13977.028663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13977.062968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13977.190579] Lustre: lustre-MDT0000: new disk, initializing [13977.347454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13977.369655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13982.341375] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13993.524569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13993.669244] Lustre: 444337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13993.777047] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13993.782315] Lustre: Skipped 1 previous similar message [13993.850740] Lustre: lustre-MDT0001: new disk, initializing [13993.976464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13994.029056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13994.054953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13999.167556] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14008.805060] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14014.447389] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14021.450916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14021.767337] Lustre: lustre-OST0000: new disk, initializing [14021.772803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14021.783539] Lustre: 446307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14021.908647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14028.281949] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14028.869327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14028.877507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14029.020181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14037.814056] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14044.641983] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14044.648048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14044.665483] Lustre: Skipped 2 previous similar messages [14044.673643] Lustre: Skipped 1 previous similar message [14049.760705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14049.768068] Lustre: Skipped 2 previous similar messages [14050.418760] Lustre: server umount lustre-MDT0000 complete [14054.072247] LustreError: 444328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785090744 with bad export cookie 7355371898163687254 [14054.074287] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14054.080432] LustreError: 444328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14054.379527] Lustre: server umount lustre-MDT0001 complete [14068.584956] Lustre: server umount lustre-OST0000 complete [14079.658698] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:32:49 (1785090769) [14088.435752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14088.955148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14094.289733] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14094.308224] LustreError: 448333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14094.326609] LustreError: 448333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14099.424807] LustreError: 448332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14104.142929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14104.705505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14109.618373] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14116.874634] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14123.166464] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14142.640269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14142.825931] Lustre: Found index 0 for lustre-OST0000, updating log [14142.961055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14148.767215] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14156.172226] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14161.395514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14166.497663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14166.506639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14166.520750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14166.528563] Lustre: Skipped 1 previous similar message [14166.540206] Lustre: Skipped 1 previous similar message [14167.855122] Lustre: server umount lustre-OST0000 complete [14183.909954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14183.916044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14183.929469] Lustre: Skipped 1 previous similar message [14183.942681] Lustre: Skipped 3 previous similar messages [14189.025030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14189.031532] Lustre: Skipped 1 previous similar message [14189.390561] Lustre: server umount lustre-MDT0000 complete [14193.125897] LustreError: 448332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14193.145920] LustreError: 448332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14193.198297] LustreError: 448312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785090883 with bad export cookie 7355371898163688647 [14193.202556] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14193.218995] LustreError: 448312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14193.672279] Lustre: server umount lustre-MDT0001 complete [14202.524819] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:34:52 (1785090892) [14208.735395] LNet: 1 local NIs in recovery (showing 1): 192.168.204.101@tcp [14822.047847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14822.425830] LustreError: 454746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14822.488183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14827.061440] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14827.489728] LustreError: 454747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14832.615333] LustreError: 454746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14835.126992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14835.631794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14840.301502] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14848.478552] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14854.434771] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14872.287494] LNet: 1 local NIs in recovery (showing 1): 192.168.204.101@tcp [14875.270529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14875.639581] Lustre: Found index 0 for lustre-OST0000, updating log [14876.271577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14884.464089] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14892.554956] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14894.793263] Lustre: server umount lustre-OST0000 complete [14898.798515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14898.815955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14898.820973] Lustre: Skipped 1 previous similar message [14899.875589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14901.922357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14901.925261] Lustre: Skipped 4 previous similar messages [14905.140931] Lustre: server umount lustre-MDT0000 complete [14906.033856] LustreError: 454746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14906.063366] LustreError: 454746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14909.409310] LustreError: 454727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785091599 with bad export cookie 7355371898163689732 [14909.416342] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14909.421908] LustreError: 454727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14910.042636] Lustre: server umount lustre-MDT0001 complete [14932.708568] LNet: 1 local NIs in recovery (showing 1): 192.168.204.101@tcp [14943.020261] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [14953.591440] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [14993.654486] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:48:03 (1785091683) [15012.841210] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [15015.499458] Key type lgssc unregistered [15015.832211] LNet: 461359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15015.840931] LNetError: 461359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15136.696628] LNet: Removed LNI 192.168.204.101@tcp [15137.964580] Key type .llcrypt unregistered [15137.968144] Key type ._llcrypt unregistered [15157.547701] Key type ._llcrypt registered [15157.552685] Key type .llcrypt registered [15157.655371] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [15158.929351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15159.022179] alg: No test for adler32 (adler32-zlib) [15160.142087] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [15160.379847] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [15162.143396] Key type lgssc registered [15163.983989] Lustre: Echo OBD driver; http://www.lustre.org/ [15169.204937] LNet: Added LNI 192.168.204.101@tcp100 [8/256/0/180] [15170.402225] LNet: Added LNI 192.168.204.101@tcp101 [8/256/0/180] [15171.641361] LNet: Added LNI 192.168.204.101@tcp102 [8/256/0/180] [15172.784873] LNet: Added LNI 192.168.204.101@tcp103 [8/256/0/180] [15173.949803] LNet: Added LNI 192.168.204.101@tcp104 [8/256/0/180] [15174.811021] LNet: Added LNI 192.168.204.101@tcp105 [8/256/0/180] [15175.728199] LNet: Added LNI 192.168.204.101@tcp106 [8/256/0/180] [15176.861869] LNet: Added LNI 192.168.204.101@tcp107 [8/256/0/180] [15177.696144] LNet: Added LNI 192.168.204.101@tcp108 [8/256/0/180] [15178.653547] LNet: Added LNI 192.168.204.101@tcp109 [8/256/0/180] [15179.532055] LNet: Added LNI 192.168.204.101@tcp110 [8/256/0/180] [15180.551728] LNet: Added LNI 192.168.204.101@tcp111 [8/256/0/180] [15181.616112] LNet: Added LNI 192.168.204.101@tcp112 [8/256/0/180] [15182.701620] LNet: Added LNI 192.168.204.101@tcp113 [8/256/0/180] [15183.582665] LNet: Added LNI 192.168.204.101@tcp114 [8/256/0/180] [15184.564905] LNet: Added LNI 192.168.204.101@tcp115 [8/256/0/180] [15185.412579] LNet: Added LNI 192.168.204.101@tcp116 [8/256/0/180] [15186.490969] LNet: Added LNI 192.168.204.101@tcp117 [8/256/0/180] [15187.330385] LNet: Added LNI 192.168.204.101@tcp118 [8/256/0/180] [15188.130551] LNet: Added LNI 192.168.204.101@tcp119 [8/256/0/180] [15189.014870] LNet: Added LNI 192.168.204.101@tcp120 [8/256/0/180] [15189.838863] LNet: Added LNI 192.168.204.101@tcp121 [8/256/0/180] [15190.744164] LNet: Added LNI 192.168.204.101@tcp122 [8/256/0/180] [15191.617677] LNet: Added LNI 192.168.204.101@tcp123 [8/256/0/180] [15192.507189] LNet: Added LNI 192.168.204.101@tcp124 [8/256/0/180] [15193.485323] LNet: Added LNI 192.168.204.101@tcp125 [8/256/0/180] [15194.396052] LNet: Added LNI 192.168.204.101@tcp126 [8/256/0/180] [15195.371391] LNet: Added LNI 192.168.204.101@tcp127 [8/256/0/180] [15196.308492] LNet: Added LNI 192.168.204.101@tcp128 [8/256/0/180] [15197.078540] LNet: Added LNI 192.168.204.101@tcp129 [8/256/0/180] [15197.899519] LNet: Added LNI 192.168.204.101@tcp130 [8/256/0/180] [15198.676614] LNet: Added LNI 192.168.204.101@tcp131 [8/256/0/180] [15199.503984] LNet: Added LNI 192.168.204.101@tcp132 [8/256/0/180] [15200.320435] LNet: Added LNI 192.168.204.101@tcp133 [8/256/0/180] [15201.105277] LNet: Added LNI 192.168.204.101@tcp134 [8/256/0/180] [15201.914147] LNet: Added LNI 192.168.204.101@tcp135 [8/256/0/180] [15208.651645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15208.712133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15209.885284] Lustre: 464394:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.101@tcp: can't add NID 192.168.204.101@tcp130: rc = -75 [15209.986130] LNetError: 464420:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.101 [15210.005319] LNetError: Protocol error connecting to 192.168.204.101@tcp121 at host 192.168.204.101:988: is it running a compatible version of Lustre? [15210.021375] LNetError: Skipped 9 previous similar messages [15210.136447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15210.168320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15210.308420] Lustre: lustre-MDT0000: new disk, initializing [15210.405231] Lustre: 464450:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.101@tcp: can't add NID 192.168.204.101@tcp130: rc = -75 [15210.420378] Lustre: 464450:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15210.443638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15210.469787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15210.594948] LNet: 3 local NIs in recovery (showing 3): 192.168.204.101@tcp127, 192.168.204.101@tcp128, 192.168.204.101@tcp121 [15211.040205] LNetError: 464414:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.101 [15211.061633] LNetError: 464414:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [15211.074841] LNetError: Protocol error connecting to 192.168.204.101@tcp126 at host 192.168.204.101:988: is it running a compatible version of Lustre? [15213.665194] LNetError: 464420:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.101 [15213.682094] LNetError: 464420:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [15213.687655] LNetError: Protocol error connecting to 192.168.204.101@tcp127 at host 192.168.204.101:988: is it running a compatible version of Lustre? [15213.695379] LNetError: Skipped 3 previous similar messages [15214.153610] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [15217.760331] LNetError: 464414:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.101 [15217.765235] LNetError: Protocol error connecting to 192.168.204.101@tcp127 at host 192.168.204.101:988: is it running a compatible version of Lustre? [15224.260764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15224.390733] Lustre: 464411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15224.441414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15224.445171] Lustre: Skipped 1 previous similar message [15224.511757] Lustre: lustre-MDT0001: new disk, initializing [15224.538700] Lustre: 465214:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.101@tcp: can't add NID 192.168.204.101@tcp130: rc = -75 [15224.544985] Lustre: 465214:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15224.648132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15224.701182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15224.715392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15225.954635] LNetError: 464419:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.101 [15225.975126] LNetError: Protocol error connecting to 192.168.204.101@tcp127 at host 192.168.204.101:988: is it running a compatible version of Lustre? [15228.935220] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [15238.490626] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15240.291531] LNetError: 464415:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.101 [15240.316540] LNetError: Protocol error connecting to 192.168.204.101@tcp124 at host 192.168.204.101:988: is it running a compatible version of Lustre? [15245.550453] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15255.062447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15255.751995] Lustre: lustre-OST0000: new disk, initializing [15255.758267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15255.783902] Lustre: 466386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15256.011741] Lustre: 466391:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.101@tcp: can't add NID 192.168.204.101@tcp130: rc = -75 [15256.020384] Lustre: 466391:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [15256.110392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15256.673133] LNetError: 464415:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.101 [15256.697907] LNetError: 464415:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15256.715477] LNetError: Protocol error connecting to 192.168.204.101@tcp109 at host 192.168.204.101:988: is it running a compatible version of Lustre? [15256.737229] LNetError: Skipped 10 previous similar messages [15265.179413] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [15265.334470] Lustre: 466744:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.101@tcp: can't add NID 192.168.204.101@tcp130: rc = -75 [15265.342698] Lustre: 466744:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15265.437950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15265.459217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15265.815741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15269.995836] LNet: 1 local NIs in recovery (showing 1): 192.168.204.101@tcp127 [15278.495122] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15291.360180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15291.375947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15291.380412] LustreError: Skipped 1 previous similar message [15291.381267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15291.433891] Lustre: Skipped 1 previous similar message [15294.568143] LNetError: 461853:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.101 [15294.585261] LNetError: 461853:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 18 previous similar messages [15294.593206] LNetError: Protocol error connecting to 192.168.204.101@tcp109 at host 192.168.204.101:988: is it running a compatible version of Lustre? [15294.602335] LNetError: Skipped 18 previous similar messages [15294.635814] Lustre: server umount lustre-OST0000 complete [15301.620031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15301.629422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15301.641283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15301.641292] Lustre: Skipped 1 previous similar message [15305.213839] Lustre: server umount lustre-MDT0000 complete [15309.449290] LustreError: 465213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785091999 with bad export cookie 7788305739069313617 [15309.455974] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15309.460318] LustreError: 465213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15309.904309] Lustre: server umount lustre-MDT0001 complete [15323.639198] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [15326.685741] Key type lgssc unregistered [15327.121847] LNet: 468301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15327.135684] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15327.154121] LNet: Removed LNI 192.168.204.101@tcp [15328.223173] LNet: Removed LNI 192.168.204.101@tcp100 [15328.225699] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15328.237158] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15328.249638] LNet: Removed LNI 192.168.204.101@tcp101 [15329.311162] LNet: Removed LNI 192.168.204.101@tcp102 [15329.313169] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15329.320942] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15329.329632] LNet: Removed LNI 192.168.204.101@tcp103 [15330.335311] LNet: Removed LNI 192.168.204.101@tcp104 [15331.359533] LNet: Removed LNI 192.168.204.101@tcp105 [15331.372457] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15331.381333] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15331.400410] LNet: Removed LNI 192.168.204.101@tcp106 [15332.447177] LNet: Removed LNI 192.168.204.101@tcp107 [15333.471704] LNet: Removed LNI 192.168.204.101@tcp108 [15334.495501] LNet: Removed LNI 192.168.204.101@tcp109 [15335.519180] LNet: Removed LNI 192.168.204.101@tcp110 [15335.522559] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15335.532238] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15335.553665] LNet: Removed LNI 192.168.204.101@tcp111 [15336.607165] LNet: Removed LNI 192.168.204.101@tcp112 [15337.631624] LNet: Removed LNI 192.168.204.101@tcp113 [15338.655327] LNet: Removed LNI 192.168.204.101@tcp114 [15339.744312] LNet: Removed LNI 192.168.204.101@tcp115 [15340.831512] LNet: Removed LNI 192.168.204.101@tcp116 [15341.857626] LNet: Removed LNI 192.168.204.101@tcp117 [15342.879227] LNet: Removed LNI 192.168.204.101@tcp118 [15343.903170] LNet: Removed LNI 192.168.204.101@tcp119 [15343.910418] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15343.932399] LNetError: 468301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15343.951649] LNet: Removed LNI 192.168.204.101@tcp120 [15344.993214] LNet: Removed LNI 192.168.204.101@tcp121 [15346.015122] LNet: Removed LNI 192.168.204.101@tcp122 [15347.039672] LNet: Removed LNI 192.168.204.101@tcp123 [15348.063234] LNet: Removed LNI 192.168.204.101@tcp124 [15349.087151] LNet: Removed LNI 192.168.204.101@tcp125 [15350.111430] LNet: Removed LNI 192.168.204.101@tcp126 [15350.116183] LNet: Removed LNI 192.168.204.101@tcp127 [15351.135178] LNet: Removed LNI 192.168.204.101@tcp128 [15352.161762] LNet: Removed LNI 192.168.204.101@tcp129 [15352.164226] LNet: Removed LNI 192.168.204.101@tcp130 [15352.167089] LNet: Removed LNI 192.168.204.101@tcp131 [15352.170283] LNet: Removed LNI 192.168.204.101@tcp132 [15352.173226] LNet: Removed LNI 192.168.204.101@tcp133 [15352.175542] LNet: Removed LNI 192.168.204.101@tcp134 [15352.188709] LNet: Removed LNI 192.168.204.101@tcp135 [15353.078517] Key type .llcrypt unregistered [15353.081288] Key type ._llcrypt unregistered [15376.042969] Key type ._llcrypt registered [15376.046051] Key type .llcrypt registered [15376.216630] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [15393.499320] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [15394.787559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15394.846619] alg: No test for adler32 (adler32-zlib) [15395.890787] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [15396.216586] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [15397.983311] Key type lgssc registered [15399.152670] Lustre: Echo OBD driver; http://www.lustre.org/ [15435.039678] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:55:24 (1785092124) [15454.630128] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [15457.458208] Key type lgssc unregistered [15457.727887] LNet: 472389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15457.736724] LNetError: 472389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15457.753113] LNet: Removed LNI 192.168.204.101@tcp [15458.573546] Key type .llcrypt unregistered [15458.575220] Key type ._llcrypt unregistered [15478.686712] Key type ._llcrypt registered [15478.690837] Key type .llcrypt registered [15478.843465] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [15480.302407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15480.393379] alg: No test for adler32 (adler32-zlib) [15481.609082] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [15482.048040] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [15483.783167] Key type lgssc registered [15484.977422] Lustre: Echo OBD driver; http://www.lustre.org/ [15495.369248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15495.422447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15496.653970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15496.690449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15496.781650] Lustre: lustre-MDT0000: new disk, initializing [15496.860952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15496.878434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15501.599737] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [15513.727520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15513.898211] Lustre: 473607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15514.015749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15514.020308] Lustre: Skipped 1 previous similar message [15514.122510] Lustre: lustre-MDT0001: new disk, initializing [15514.223433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15514.271076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15514.285282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15519.485715] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [15530.105599] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15536.949459] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15544.309824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15544.562676] Lustre: lustre-OST0000: new disk, initializing [15544.568328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15544.579621] Lustre: 475573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15544.687528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15551.487270] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [15551.553157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15551.566059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15551.759509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15561.605197] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15565.909022] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing lnet_if_list [15571.530960] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15571.569566] LNet: Added LNI 192.168.204.101@tcp10 [8/256/0/180] [15577.739392] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15577.783643] LNet: Added LNI 192.168.204.101@tcp11 [8/256/0/180] [15582.213901] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15582.264466] LNet: Added LNI 192.168.204.101@tcp12 [8/256/0/180] [15586.772970] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15586.815355] LNet: Added LNI 192.168.204.101@tcp13 [8/256/0/180] [15591.531537] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15591.577394] LNet: Added LNI 192.168.204.101@tcp14 [8/256/0/180] [15596.183760] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15596.209529] LNet: Added LNI 192.168.204.101@tcp15 [8/256/0/180] [15601.154426] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15601.187440] LNet: Added LNI 192.168.204.101@tcp16 [8/256/0/180] [15605.990586] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15606.021728] LNet: Added LNI 192.168.204.101@tcp17 [8/256/0/180] [15611.345745] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15611.384463] LNet: Added LNI 192.168.204.101@tcp18 [8/256/0/180] [15617.004343] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15617.048400] LNet: Added LNI 192.168.204.101@tcp19 [8/256/0/180] [15623.256326] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15623.304359] LNet: Added LNI 192.168.204.101@tcp20 [8/256/0/180] [15630.752466] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15630.776936] LNet: Added LNI 192.168.204.101@tcp21 [8/256/0/180] [15639.788538] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15639.827120] LNet: Added LNI 192.168.204.101@tcp22 [8/256/0/180] [15646.580809] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15646.608859] LNet: Added LNI 192.168.204.101@tcp23 [8/256/0/180] [15653.293295] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15653.321482] LNet: Added LNI 192.168.204.101@tcp24 [8/256/0/180] [15660.689988] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15660.717751] LNet: Added LNI 192.168.204.101@tcp25 [8/256/0/180] [15668.095567] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15668.167984] LNet: Added LNI 192.168.204.101@tcp26 [8/256/0/180] [15674.754115] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15674.788819] LNet: Added LNI 192.168.204.101@tcp27 [8/256/0/180] [15681.628102] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15681.674355] LNet: Added LNI 192.168.204.101@tcp28 [8/256/0/180] [15688.571305] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15688.600652] LNet: Added LNI 192.168.204.101@tcp29 [8/256/0/180] [15695.791653] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15695.826967] LNet: Added LNI 192.168.204.101@tcp30 [8/256/0/180] [15703.538266] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15703.560355] LNet: Added LNI 192.168.204.101@tcp31 [8/256/0/180] [15710.943741] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15710.976643] LNet: Added LNI 192.168.204.101@tcp32 [8/256/0/180] [15718.573565] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15718.655684] LNet: Added LNI 192.168.204.101@tcp33 [8/256/0/180] [15726.196254] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15726.230697] LNet: Added LNI 192.168.204.101@tcp34 [8/256/0/180] [15733.842740] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15733.908089] LNet: Added LNI 192.168.204.101@tcp35 [8/256/0/180] [15741.138987] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15741.176670] LNet: Added LNI 192.168.204.101@tcp36 [8/256/0/180] [15748.839667] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15748.882377] LNet: Added LNI 192.168.204.101@tcp37 [8/256/0/180] [15757.175681] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15757.230488] LNet: Added LNI 192.168.204.101@tcp38 [8/256/0/180] [15764.817198] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15764.840723] LNet: Added LNI 192.168.204.101@tcp39 [8/256/0/180] [15772.719624] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15772.747873] LNet: Added LNI 192.168.204.101@tcp40 [8/256/0/180] [15781.017083] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15781.047102] LNet: Added LNI 192.168.204.101@tcp41 [8/256/0/180] [15788.473388] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15788.524408] LNet: Added LNI 192.168.204.101@tcp42 [8/256/0/180] [15797.177849] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15797.222927] LNet: Added LNI 192.168.204.101@tcp43 [8/256/0/180] [15806.567317] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15806.632830] LNet: Added LNI 192.168.204.101@tcp44 [8/256/0/180] [15815.127989] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15815.156681] LNet: Added LNI 192.168.204.101@tcp45 [8/256/0/180] [15823.746916] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15823.779061] LNet: Added LNI 192.168.204.101@tcp46 [8/256/0/180] [15831.699527] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15831.734880] LNet: Added LNI 192.168.204.101@tcp47 [8/256/0/180] [15840.028827] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15840.064872] LNet: Added LNI 192.168.204.101@tcp48 [8/256/0/180] [15848.071924] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15848.099646] LNet: Added LNI 192.168.204.101@tcp49 [8/256/0/180] [15857.297103] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15857.325549] LNet: Added LNI 192.168.204.101@tcp50 [8/256/0/180] [15877.600576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15877.605562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15877.615602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15879.650985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15879.651443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15879.662457] Lustre: Skipped 1 previous similar message [15879.672629] Lustre: Skipped 2 previous similar messages [15881.482292] Lustre: server umount lustre-MDT0000 complete [15885.102489] LustreError: 473597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785092575 with bad export cookie 4893445328771830790 [15885.103729] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15885.117680] LustreError: 473597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15885.377947] Lustre: server umount lustre-MDT0001 complete [15903.637959] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [15906.080088] Lustre: 472879:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785092580/real 1785092580] req@0000000068c6ee06 x1871804809529984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785092596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15906.101927] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15907.295331] Lustre: 472878:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785092581/real 1785092581] req@00000000ad1cfb62 x1871804809530112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785092597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15911.391104] Lustre: 472880:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785092585/real 1785092585] req@0000000045ef868b x1871804809530496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785092601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15912.389298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15916.447168] Lustre: 472878:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785092590/real 1785092590] req@000000000e1c5d26 x1871804809530624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785092606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15916.477179] Lustre: 472878:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15921.631248] Lustre: 472878:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785092595/real 1785092595] req@00000000dacb64df x1871804809530752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785092611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15930.719487] LustreError: 472877:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000855e675c x1871804809531392/t0(0) o250->MGC192.168.204.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15931.044785] LustreError: 475566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15931.070921] LustreError: 475566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15931.140173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15932.128395] LustreError: 499832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15932.154350] LustreError: 499832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15935.204573] LustreError: 499829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15935.210381] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15935.224151] LustreError: 499829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15935.539991] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [15940.581460] LustreError: 499832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15940.616258] LustreError: 499832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15943.901807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15944.210748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15948.741555] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [15949.291404] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15955.566706] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15961.047916] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15982.561411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15982.572159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15982.582511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15985.123074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15990.247025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15990.255792] Lustre: Skipped 2 previous similar messages [15994.336578] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15994.511500] Lustre: server umount lustre-OST0000 complete [16000.480672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16000.484402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16000.513914] Lustre: Skipped 1 previous similar message [16000.515026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16000.537334] Lustre: Skipped 1 previous similar message [16004.686558] Lustre: server umount lustre-MDT0000 complete [16005.605217] LustreError: 501722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16005.613496] LustreError: 501722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16008.326728] LustreError: 473598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785092698 with bad export cookie 4893445328771833478 [16008.336305] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16008.343980] LustreError: 473598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16008.725957] Lustre: server umount lustre-MDT0001 complete [16018.136677] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [16020.512952] Key type lgssc unregistered [16020.857917] LNet: 502633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16020.871260] LNetError: 502633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16021.919206] LNet: Removed LNI 192.168.204.101@tcp [16021.925501] LNetError: 502633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16021.940985] LNet: Removed LNI 192.168.204.101@tcp10 [16021.948778] LNet: Removed LNI 192.168.204.101@tcp11 [16021.952335] LNet: Removed LNI 192.168.204.101@tcp12 [16021.958474] LNet: Removed LNI 192.168.204.101@tcp13 [16021.963733] LNet: Removed LNI 192.168.204.101@tcp14 [16021.970498] LNet: Removed LNI 192.168.204.101@tcp15 [16021.979357] LNet: Removed LNI 192.168.204.101@tcp16 [16021.983889] LNet: Removed LNI 192.168.204.101@tcp17 [16021.987692] LNet: Removed LNI 192.168.204.101@tcp18 [16021.994883] LNet: Removed LNI 192.168.204.101@tcp19 [16021.998596] LNet: Removed LNI 192.168.204.101@tcp20 [16022.005113] LNet: Removed LNI 192.168.204.101@tcp21 [16022.009099] LNet: Removed LNI 192.168.204.101@tcp22 [16022.012653] LNet: Removed LNI 192.168.204.101@tcp23 [16022.017149] LNet: Removed LNI 192.168.204.101@tcp24 [16022.019377] LNet: Removed LNI 192.168.204.101@tcp25 [16022.022839] LNet: Removed LNI 192.168.204.101@tcp26 [16022.026310] LNet: Removed LNI 192.168.204.101@tcp27 [16022.030132] LNet: Removed LNI 192.168.204.101@tcp28 [16022.042162] LNet: Removed LNI 192.168.204.101@tcp29 [16022.048452] LNet: Removed LNI 192.168.204.101@tcp30 [16022.052265] LNet: Removed LNI 192.168.204.101@tcp31 [16022.059390] LNet: Removed LNI 192.168.204.101@tcp32 [16022.065427] LNet: Removed LNI 192.168.204.101@tcp33 [16022.071974] LNet: Removed LNI 192.168.204.101@tcp34 [16022.080501] LNet: Removed LNI 192.168.204.101@tcp35 [16022.083639] LNet: Removed LNI 192.168.204.101@tcp36 [16022.086968] LNet: Removed LNI 192.168.204.101@tcp37 [16022.092914] LNet: Removed LNI 192.168.204.101@tcp38 [16022.097423] LNet: Removed LNI 192.168.204.101@tcp39 [16022.101837] LNet: Removed LNI 192.168.204.101@tcp40 [16022.104140] LNet: Removed LNI 192.168.204.101@tcp41 [16022.106791] LNet: Removed LNI 192.168.204.101@tcp42 [16022.109983] LNet: Removed LNI 192.168.204.101@tcp43 [16022.114238] LNet: Removed LNI 192.168.204.101@tcp44 [16022.116802] LNet: Removed LNI 192.168.204.101@tcp45 [16022.119780] LNet: Removed LNI 192.168.204.101@tcp46 [16022.125958] LNet: Removed LNI 192.168.204.101@tcp47 [16022.128636] LNet: Removed LNI 192.168.204.101@tcp48 [16022.131281] LNet: Removed LNI 192.168.204.101@tcp49 [16022.142075] LNet: Removed LNI 192.168.204.101@tcp50 [16022.789215] Key type .llcrypt unregistered [16022.790902] Key type ._llcrypt unregistered [16041.780385] Key type ._llcrypt registered [16041.784245] Key type .llcrypt registered [16041.944680] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [16056.725761] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [16057.655374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16057.767903] alg: No test for adler32 (adler32-zlib) [16058.809316] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [16059.030035] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [16060.727174] Key type lgssc registered [16061.910603] Lustre: Echo OBD driver; http://www.lustre.org/ [16094.680990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16094.709083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16095.884910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16095.909385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16095.979545] Lustre: lustre-MDT0000: new disk, initializing [16096.041734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16096.060181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16099.809561] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16109.288739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16109.416717] Lustre: 505825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16109.460875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16109.464463] Lustre: Skipped 1 previous similar message [16109.525861] Lustre: lustre-MDT0001: new disk, initializing [16109.590064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16109.631273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16109.643568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16113.775861] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16123.343410] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16129.376343] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16137.285241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16137.490328] Lustre: *** cfs_fail_loc=90f, val=0*** [16137.566667] Lustre: lustre-OST0000: new disk, initializing [16137.570384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16137.580920] Lustre: 507842:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16137.684492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16143.247981] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16146.977521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16146.985167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16147.069294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16151.482493] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16162.282225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16162.289379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16162.290559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16162.303863] Lustre: Skipped 1 previous similar message [16162.316666] Lustre: Skipped 1 previous similar message [16164.100421] Lustre: server umount lustre-OST0000 complete [16170.982078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16170.993409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16171.010250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16172.513717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16172.514677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16177.635570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16177.644591] Lustre: Skipped 2 previous similar messages [16182.239156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16182.526532] Lustre: server umount lustre-MDT0000 complete [16186.463025] LustreError: 505818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785092876 with bad export cookie 13976381031341124106 [16186.471313] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16186.484167] LustreError: 505818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16186.877671] Lustre: server umount lustre-MDT0001 complete [16197.081946] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [16199.577282] Key type lgssc unregistered [16199.864768] LNet: 509750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16199.878126] LNetError: 509750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16200.933691] LNet: Removed LNI 192.168.204.101@tcp [16201.854801] Key type .llcrypt unregistered [16201.856857] Key type ._llcrypt unregistered [16219.336451] Key type ._llcrypt registered [16219.338557] Key type .llcrypt registered [16219.456068] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [16220.635971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16220.659992] alg: No test for adler32 (adler32-zlib) [16221.654279] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [16221.874447] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [16223.511190] Key type lgssc registered [16224.358534] Lustre: Echo OBD driver; http://www.lustre.org/ [16230.359186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16230.378435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16231.722365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16235.503428] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16241.123669] LustreError: 511025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16241.141164] LustreError: 511025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16242.750878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16243.058241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16246.928166] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16252.775765] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16256.989929] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16262.369492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16262.607320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16267.205918] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16274.532225] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16283.957775] loop0: detected capacity change from 0 to 204800000 [16284.112196] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16284.120986] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16284.137457] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16289.275577] loop0: detected capacity change from 0 to 204800000 [16289.422753] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16289.435655] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16289.446655] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16293.531699] loop0: detected capacity change from 0 to 204800000 [16298.343749] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16298.494846] Lustre: fs15246-OST0000: new disk, initializing [16298.501604] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16298.514833] Lustre: 514620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16298.581267] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16303.164682] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16311.215586] loop1: detected capacity change from 0 to 204800000 [16315.117716] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16315.188527] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16315.241059] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16315.306488] Lustre: fs15246-MDT0000: new disk, initializing [16315.416338] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16315.431644] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16315.493659] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16316.512648] Lustre: 515743:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d29d4ecf x1871805585335808/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785093022 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16318.655420] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16330.743234] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16330.755573] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16330.779108] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16335.849786] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16335.868486] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16340.962993] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16341.106886] Lustre: server umount fs15246-MDT0000 complete [16346.083656] LustreError: 511019:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16347.477701] Lustre: server umount fs15246-OST0000 complete [16356.328507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16356.331340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16356.340386] Lustre: Skipped 1 previous similar message [16356.358189] Lustre: Skipped 1 previous similar message [16361.442901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16361.450476] Lustre: Skipped 1 previous similar message [16362.242899] Lustre: server umount lustre-OST0000 complete [16366.048430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16366.053799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16366.067626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16376.801775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16376.808955] Lustre: Skipped 4 previous similar messages [16379.359176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16379.529426] Lustre: server umount lustre-MDT0000 complete [16381.822829] LustreError: 515737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785093072 with bad export cookie 15809194341060137505 [16381.825363] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16381.829184] LustreError: 515737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16382.045976] Lustre: server umount lustre-MDT0001 complete [16388.783579] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [16391.099136] Key type lgssc unregistered [16391.385610] LNet: 518836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16391.394411] LNetError: 518836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16391.411170] LNet: Removed LNI 192.168.204.101@tcp [16392.105689] Key type .llcrypt unregistered [16392.108205] Key type ._llcrypt unregistered [16404.623315] Key type ._llcrypt registered [16404.626154] Key type .llcrypt registered [16404.697497] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [16405.255867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16405.271902] alg: No test for adler32 (adler32-zlib) [16406.256967] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [16406.436804] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [16408.087410] Key type lgssc registered [16408.862622] Lustre: Echo OBD driver; http://www.lustre.org/ [16437.984233] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [16445.853967] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [16471.593874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16471.625622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16472.817636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16472.861806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16472.959682] Lustre: lustre-MDT0000: new disk, initializing [16473.072123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16473.118475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16476.712076] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16484.089399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16484.187563] Lustre: 523275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16484.253642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16484.262304] Lustre: Skipped 1 previous similar message [16484.364176] Lustre: lustre-MDT0001: new disk, initializing [16484.443839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16484.472398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16484.480740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16487.149915] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16494.162072] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16498.130884] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16503.015302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16503.273864] Lustre: lustre-OST0000: new disk, initializing [16503.278393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16503.285603] Lustre: 525243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16503.356264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16508.325769] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [16512.557521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16512.565750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16512.655375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16515.625494] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16518.615819] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16518.766764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16522.253803] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16522.431854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16527.840236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16527.848693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16527.861864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16528.356736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16528.358537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16528.410885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16528.414714] Lustre: Skipped 1 previous similar message [16530.007496] Lustre: server umount lustre-OST0000 complete [16533.474724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16533.487414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16533.491598] Lustre: Skipped 1 previous similar message [16538.598908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16538.606567] Lustre: Skipped 1 previous similar message [16543.713121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16543.717032] Lustre: Skipped 1 previous similar message [16547.295187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16547.437792] Lustre: server umount lustre-MDT0000 complete [16550.316039] LustreError: 523268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785093240 with bad export cookie 2450984964462340808 [16550.321124] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16550.330902] LustreError: 523268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16550.637497] Lustre: server umount lustre-MDT0001 complete [16566.537123] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [16572.701577] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [16593.649480] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16263 sec ========= 15:14:43 (1785093283) [16594.997614] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:14:45 (1785093285) === [16597.551697] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:14:47 (1785093287) === [16624.505445] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [16626.571293] Key type lgssc unregistered [16626.810728] LNet: 531885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16626.821163] LNetError: 531885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16626.835592] LNet: Removed LNI 192.168.204.101@tcp [16627.468663] Key type .llcrypt unregistered [16627.470800] Key type ._llcrypt unregistered