[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473442271 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003193] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013080] *** VALIDATE selinux *** [ 0.021443] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026444] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030052] *** VALIDATE tmpfs *** [ 0.031524] *** VALIDATE proc *** [ 0.032254] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034347] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035181] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037039] Spectre V2 : User space: Vulnerable [ 0.038014] Speculative Store Bypass: Vulnerable [ 0.041683] debug: unmapping init [mem 0xffffffff88859000-0xffffffff88860fff] [ 0.043993] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044713] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045034] ... version: 2 [ 0.046018] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048018] ... value mask: 0000ffffffffffff [ 0.049021] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052350] rcu: Hierarchical SRCU implementation. [ 0.054569] smp: Bringing up secondary CPUs ... [ 0.055635] x86: Booting SMP configuration: [ 0.056024] .... node #0, CPUs: #1 #2 #3 [ 0.059610] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.210322] node 0 deferred pages initialised in 146ms [ 0.214223] devtmpfs: initialized [ 0.215271] x86/mm: Memory block size: 128MB [ 0.218820] gcov: version magic: 0x41383552 [ 0.221194] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.224076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.226365] pinctrl core: initialized pinctrl subsystem [ 0.228183] [ 0.228813] ************************************************************* [ 0.231010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.233011] ** ** [ 0.235009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.237012] ** ** [ 0.240014] ** This means that this kernel is built to expose internal ** [ 0.242012] ** IOMMU data structures, which may compromise security on ** [ 0.245015] ** your system. ** [ 0.248018] ** ** [ 0.251013] ** If you see this message and you are not debugging the ** [ 0.254015] ** kernel, report this immediately to your vendor! ** [ 0.256011] ** ** [ 0.259014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261013] ************************************************************* [ 0.264755] NET: Registered protocol family 16 [ 0.267402] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.270068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.272066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.276450] cpuidle: using governor menu [ 0.278480] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.283028] PCI: Using configuration type 1 for base access [ 0.285110] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.295116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.297033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.301096] cryptd: max_cpu_qlen set to 1000 [ 0.304246] ACPI: Added _OSI(Module Device) [ 0.306018] ACPI: Added _OSI(Processor Device) [ 0.307021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.309015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.314108] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.321460] ACPI: Interpreter enabled [ 0.323053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.324016] ACPI: Using IOAPIC for interrupt routing [ 0.326174] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.330455] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.342059] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.344070] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.347044] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.355092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.360507] acpiphp: Slot [2] registered [ 0.363289] acpiphp: Slot [5] registered [ 0.365222] acpiphp: Slot [6] registered [ 0.366184] acpiphp: Slot [7] registered [ 0.368119] acpiphp: Slot [8] registered [ 0.370173] acpiphp: Slot [9] registered [ 0.371160] acpiphp: Slot [10] registered [ 0.373173] acpiphp: Slot [3] registered [ 0.375142] acpiphp: Slot [4] registered [ 0.377150] acpiphp: Slot [11] registered [ 0.378157] acpiphp: Slot [12] registered [ 0.380124] acpiphp: Slot [13] registered [ 0.381116] acpiphp: Slot [14] registered [ 0.383200] acpiphp: Slot [15] registered [ 0.385146] acpiphp: Slot [16] registered [ 0.387213] acpiphp: Slot [17] registered [ 0.389214] acpiphp: Slot [18] registered [ 0.391341] acpiphp: Slot [19] registered [ 0.393317] acpiphp: Slot [20] registered [ 0.397208] acpiphp: Slot [21] registered [ 0.398183] acpiphp: Slot [22] registered [ 0.400296] acpiphp: Slot [23] registered [ 0.402275] acpiphp: Slot [24] registered [ 0.404188] acpiphp: Slot [25] registered [ 0.406177] acpiphp: Slot [26] registered [ 0.408151] acpiphp: Slot [27] registered [ 0.410170] acpiphp: Slot [28] registered [ 0.413168] acpiphp: Slot [29] registered [ 0.414131] acpiphp: Slot [30] registered [ 0.415179] acpiphp: Slot [31] registered [ 0.416119] PCI host bridge to bus 0000:00 [ 0.417034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.419050] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.422034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.428035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.432032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.436212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.442098] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.445468] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.459032] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.465050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.468027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.471028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.474027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.477775] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.481846] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.485067] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.489000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.495030] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.508036] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.512032] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.518996] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.526022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.533020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.550021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.560960] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.570026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.577027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.596023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.612129] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.624028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.634019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.661000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.676893] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.685027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.692033] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.713000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.725913] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.732027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.745030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.769023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.778466] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.789021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.796024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.815022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.825921] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.829474] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.831396] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.834458] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.838244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.845129] iommu: Default domain type: Passthrough [ 0.847526] SCSI subsystem initialized [ 0.849168] ACPI: bus type USB registered [ 0.850117] usbcore: registered new interface driver usbfs [ 0.852141] usbcore: registered new interface driver hub [ 0.855114] usbcore: registered new device driver usb [ 0.857317] pps_core: LinuxPPS API ver. 1 registered [ 0.859017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.862209] PTP clock support registered [ 0.865060] EDAC MC: Ver: 3.0.0 [ 0.867158] PCI: Using ACPI for IRQ routing [ 0.869974] NetLabel: Initializing [ 0.871012] NetLabel: domain hash size = 128 [ 0.873012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.875086] NetLabel: unlabeled traffic allowed by default [ 0.877114] vgaarb: loaded [ 0.879291] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.881016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.886332] clocksource: Switched to clocksource kvm-clock [ 1.001465] VFS: Disk quotas dquot_6.6.0 [ 1.003714] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.006533] *** VALIDATE ramfs *** [ 1.008128] *** VALIDATE hugetlbfs *** [ 1.009947] pnp: PnP ACPI init [ 1.012590] pnp: PnP ACPI: found 6 devices [ 1.040714] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.044429] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.046954] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.049312] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.051759] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.054370] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.058093] NET: Registered protocol family 2 [ 1.060800] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.065938] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.069273] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.074884] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.079053] TCP: Hash tables configured (established 65536 bind 65536) [ 1.082582] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.086480] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.089649] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.093336] NET: Registered protocol family 1 [ 1.097335] RPC: Registered named UNIX socket transport module. [ 1.100223] RPC: Registered udp transport module. [ 1.102177] RPC: Registered tcp transport module. [ 1.104478] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.107153] NET: Registered protocol family 44 [ 1.108867] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.111448] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.113880] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.116456] PCI: CLS 0 bytes, default 64 [ 1.118214] Unpacking initramfs... [ 2.570488] debug: unmapping init [mem 0xffff8896fcc54000-0xffff8896fffbffff] [ 2.576810] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.579589] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.582903] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.117671] Initialise system trusted keyrings [ 3.121916] Key type blacklist registered [ 3.123991] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.134398] zbud: loaded [ 3.137591] *** VALIDATE nfs *** [ 3.139054] *** VALIDATE nfs4 *** [ 3.140824] pstore: using deflate compression [ 3.145124] Platform Keyring initialized [ 3.262486] NET: Registered protocol family 38 [ 3.264441] Key type asymmetric registered [ 3.266110] Asymmetric key parser 'x509' registered [ 3.268196] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.271619] io scheduler mq-deadline registered [ 3.273588] io scheduler kyber registered [ 3.275456] io scheduler bfq registered [ 3.277872] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.281754] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.285277] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.288765] ACPI: Power Button [PWRF] [ 3.295330] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.302259] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.314808] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.322076] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.335870] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.364381] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.393189] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.400706] Non-volatile memory driver v1.3 [ 3.402253] Linux agpgart interface v0.103 [ 3.440585] virtio_blk virtio1: [vda] 146944 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.444124] vda: detected capacity change from 0 to 75235328 [ 3.465434] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.468769] vdb: detected capacity change from 0 to 1073741824 [ 3.488247] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.491230] vdc: detected capacity change from 0 to 2621440000 [ 3.514489] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.517388] vdd: detected capacity change from 0 to 2621440000 [ 3.538863] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.544472] vde: detected capacity change from 0 to 4294967296 [ 3.562219] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.565327] vdf: detected capacity change from 0 to 4294967296 [ 3.573417] libphy: Fixed MDIO Bus: probed [ 3.579343] usbcore: registered new interface driver usbserial_generic [ 3.582358] usbserial: USB Serial support registered for generic [ 3.585132] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.589461] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.591604] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.594245] mousedev: PS/2 mouse device common for all mice [ 3.597828] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.600030] rtc_cmos 00:05: RTC can wake from S4 [ 3.607923] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.611466] rtc_cmos 00:05: registered as rtc0 [ 3.614134] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.616619] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.617750] intel_pstate: CPU model not supported [ 3.624566] hid: raw HID events driver (C) Jiri Kosina [ 3.627139] usbcore: registered new interface driver usbhid [ 3.629640] usbhid: USB HID core driver [ 3.633368] drop_monitor: Initializing network drop monitor service [ 3.635340] Initializing XFRM netlink socket [ 3.636748] NET: Registered protocol family 10 [ 3.639926] Segment Routing with IPv6 [ 3.641677] NET: Registered protocol family 17 [ 3.644373] mpls_gso: MPLS GSO support [ 3.649848] RAS: Correctable Errors collector initialized. [ 3.651184] AVX version of gcm_enc/dec engaged. [ 3.652303] AES CTR mode by8 optimization enabled [ 3.730919] sched_clock: Marking stable (3730894356, 0)->(4711777802, -980883446) [ 3.734455] registered taskstats version 1 [ 3.736716] Loading compiled-in X.509 certificates [ 3.739316] zswap: loaded using pool lzo/zbud [ 3.766057] Key type big_key registered [ 3.778927] Key type encrypted registered [ 3.780941] ima: No TPM chip found, activating TPM-bypass! [ 3.783326] ima: Allocated hash algorithm: sha1 [ 3.785438] ima: No architecture policies found [ 3.787710] evm: Initialising EVM extended attributes: [ 3.790015] evm: security.selinux [ 3.791043] evm: security.ima [ 3.791771] evm: security.capability [ 3.792686] evm: HMAC attrs: 0x1 [ 3.794950] rtc_cmos 00:05: setting system clock to 2026-08-27 13:11:11 UTC (1787836271) [ 3.802167] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 3.806032] debug: unmapping init [mem 0xffffffff88582000-0xffffffff88858fff] [ 3.814094] Write protecting the kernel read-only data: 28672k [ 3.818139] debug: unmapping init [mem 0xffffffff86c03000-0xffffffff86dfffff] [ 3.821506] debug: unmapping init [mem 0xffffffff87514000-0xffffffff875fffff] [ 3.861417] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.871453] systemd[1]: Detected virtualization kvm. [ 3.873578] systemd[1]: Detected architecture x86-64. [ 3.875589] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.905897] systemd[1]: No hostname configured. [ 3.908335] systemd[1]: Set hostname to . [ 3.910946] random: systemd: uninitialized urandom read (16 bytes read) [ 3.913584] systemd[1]: Initializing machine ID from random generator. [ 3.957973] random: ln: uninitialized urandom read (6 bytes read) [ 4.054299] random: systemd: uninitialized urandom read (16 bytes read) [ 4.056682] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.061489] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.066538] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.774456] device-mapper: uevent: version 1.0.3 [ 4.777192] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.602250] virtio_net virtio0 ens2: renamed from eth0 [ 5.633063] random: fast init done [ 5.662272] scsi host0: ata_piix [ 5.694445] scsi host1: ata_piix [ 5.712388] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.715649] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.425923] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.297652] random: crng init done [ 10.299374] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.813594] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.954609] printk: systemd: 23 output lines suppressed due to ratelimiting [ 12.203649] SELinux: Disabled at runtime. [ 12.263424] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.273743] systemd[1]: Detected virtualization kvm. [ 12.275875] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.747042] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.751620] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.756578] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.762380] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.766106] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.773413] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.778495] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ 12.871082] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.242178] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.530799] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.542201] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.721936] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.735714] EDAC sbridge: Ver: 1.1.2 [ 15.635547] Key type dns_resolver registered [ 15.943449] NFS: Registering the id_resolver key type [ 15.945482] Key type id_resolver registered [ 15.947456] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg322-server login: [ 70.227799] libcfs: loading out-of-tree module taints kernel. [ 70.305115] Key type ._llcrypt registered [ 70.306906] Key type .llcrypt registered [ 70.418628] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 89.643745] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 91.042762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.078364] alg: No test for adler32 (adler32-zlib) [ 92.581397] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 93.590879] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 95.320278] Key type lgssc registered [ 97.006233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.841846] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.084323] hrtimer: interrupt took 9338752 ns [ 160.262276] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 177.720729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 177.793670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.037985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.080724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.201907] Lustre: lustre-MDT0000: new disk, initializing [ 179.317042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.333591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.083532] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.017882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.132187] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.155287] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 201.158465] Lustre: Skipped 1 previous similar message [ 201.254447] Lustre: lustre-MDT0001: new disk, initializing [ 201.321198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.352867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 201.367760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.035439] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.733723] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.327046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.619891] Lustre: lustre-OST0000: new disk, initializing [ 220.629468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.640615] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.727694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.805618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.820565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.885865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.641791] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.620645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.764381] Lustre: lustre-OST0001: new disk, initializing [ 240.767655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.772188] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.839523] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 247.271027] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.386246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 249.401240] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 249.448563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.339429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 269.800837] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.272652] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing check_logdir /tmp/testlogs/ [ 284.279134] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing yml_node [ 289.033148] Lustre: DEBUG MARKER: Client: 2.17.57.96 [ 291.977426] Lustre: DEBUG MARKER: MDS: 2.17.57.96 [ 295.427473] Lustre: DEBUG MARKER: OSS: 2.17.57.96 [ 297.290357] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 09:16:02 EDT 2026 [ 316.497937] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 318.240309] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 334.308270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 334.320803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.336264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.355395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.358122] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.370033] Lustre: Skipped 2 previous similar messages [ 336.379224] Lustre: Skipped 2 previous similar messages [ 340.371783] Lustre: server umount lustre-MDT0000 complete [ 344.735340] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836612 with bad export cookie 13659173904800650207 [ 344.740159] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 344.749369] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 346.605278] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.605328] LustreError: 11913:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.606485] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.624093] Lustre: Skipped 1 previous similar message [ 346.658096] LustreError: 11913:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 349.668047] LustreError: 6522:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.668495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.695390] LustreError: 6522:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 349.708432] Lustre: Skipped 2 previous similar messages [ 354.793627] LustreError: 11913:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.793695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 354.823863] LustreError: 11913:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 354.829221] Lustre: Skipped 1 previous similar message [ 359.392811] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 359.821378] Lustre: server umount lustre-MDT0001 complete [ 363.621824] Lustre: server umount lustre-OST0000 complete [ 367.641208] Lustre: server umount lustre-OST0001 complete [ 375.671357] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 386.923855] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 427.107873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.402479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 427.439798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 427.521579] Lustre: lustre-MDT0000: new disk, initializing [ 427.632263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.658614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 432.641443] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 444.735805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.963969] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 445.023410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 445.027194] Lustre: Skipped 1 previous similar message [ 445.136326] Lustre: lustre-MDT0001: new disk, initializing [ 445.292669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 445.335937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 445.347564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 449.997565] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 460.238494] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 467.170966] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 474.279409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.681574] Lustre: lustre-OST0000: new disk, initializing [ 474.685982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 474.691890] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 474.826459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.006894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 476.015876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 476.194644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 482.019865] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 491.420975] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 495.968747] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 496.156545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 500.640938] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 500.894760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 506.849350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.857525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.864453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.874668] Lustre: Skipped 1 previous similar message [ 506.895981] Lustre: Skipped 1 previous similar message [ 509.167722] Lustre: server umount lustre-OST0000 complete [ 517.093260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 517.092573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.105726] LustreError: Skipped 1 previous similar message [ 517.114930] Lustre: Skipped 1 previous similar message [ 526.306836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.317634] Lustre: Skipped 5 previous similar messages [ 527.328154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 527.528177] Lustre: server umount lustre-MDT0000 complete [ 531.007586] LustreError: 15876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836798 with bad export cookie 13659173904800651740 [ 531.017624] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 531.024068] LustreError: 15876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 531.376849] Lustre: server umount lustre-MDT0001 complete [ 535.197066] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:20:00 (1787836800) [ 537.318353] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 539.444626] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:20:04 (1787836804) [ 545.814904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.267342] LustreError: 19460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.355136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.908541] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 551.396671] LustreError: 19461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.618196] LustreError: 19461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.650128] LustreError: 19461:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 560.812495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.345709] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 572.273505] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 577.248614] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 584.355351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 591.165495] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 599.254452] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 611.297762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 611.299225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.315729] LustreError: Skipped 1 previous similar message [ 611.329068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 611.347873] Lustre: Skipped 1 previous similar message [ 611.373279] Lustre: Skipped 1 previous similar message [ 613.337808] Lustre: server umount lustre-OST0000 complete [ 617.455810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.618202] Lustre: server umount lustre-MDT0000 complete [ 626.660680] LustreError: 20188:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.686529] LustreError: 20188:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 627.681216] LustreError: 20196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836895 with bad export cookie 13659173904800652622 [ 627.695578] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.701069] LustreError: 20196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 628.105638] Lustre: server umount lustre-MDT0001 complete [ 641.487367] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 644.962720] Key type lgssc unregistered [ 645.250556] LNet: 22999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.260334] LNetError: 22999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.279285] LNet: Removed LNI 192.168.203.122@tcp [ 646.441172] Key type .llcrypt unregistered [ 646.448279] Key type ._llcrypt unregistered [ 670.492274] Key type ._llcrypt registered [ 670.497412] Key type .llcrypt registered [ 670.643522] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 672.244545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.276555] alg: No test for adler32 (adler32-zlib) [ 673.368493] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 673.642738] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 675.306727] Key type lgssc registered [ 676.498264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.568351] loop: module loaded [ 684.633369] loop0: detected capacity change from 0 to 204800000 [ 684.934716] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.944304] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.227158] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.808975] loop0: detected capacity change from 0 to 204800000 [ 691.027286] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.043262] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.065954] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.659501] loop0: detected capacity change from 0 to 204800000 [ 702.708674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 702.730793] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.030708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 704.080340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 704.203832] Lustre: lustre-MDT0000: new disk, initializing [ 704.307403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 704.348934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 708.826826] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 718.565215] loop1: detected capacity change from 0 to 204800000 [ 724.261932] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 724.592246] Lustre: lustre-OST07c6: new disk, initializing [ 724.599965] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 724.608923] Lustre: Skipped 1 previous similar message [ 724.617600] Lustre: 26186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 724.710595] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 730.704466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 730.719086] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 730.766453] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 730.878188] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 736.487949] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 736.494933] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 736.498192] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 736.504689] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 736.518617] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 736.530041] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 750.358796] Lustre: 26701:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 750.364852] Lustre: 26701:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.370598] Lustre: 26701:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.376721] Lustre: 26701:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 750.382898] Lustre: 26701:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 750.388418] Lustre: 26701:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.085414] Lustre: 26701:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 755.107203] Lustre: 26701:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 755.123168] Lustre: 26701:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 755.138290] Lustre: 26701:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.151387] Lustre: 26701:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.167321] Lustre: 26701:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.185478] Lustre: 26701:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 755.194483] Lustre: 26701:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.202521] Lustre: 26701:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 755.210224] Lustre: 26701:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.224736] Lustre: 26701:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.235443] Lustre: 26701:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.470065] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 758.487050] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 758.499472] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 758.504741] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.512880] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.520356] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.525121] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 758.530571] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.540075] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 758.544913] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.554142] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.564894] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.276329] Lustre: 26701:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 763.300025] Lustre: 26701:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 763.315947] Lustre: 26701:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.333307] Lustre: 26701:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.340097] Lustre: 26701:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.356209] Lustre: 26701:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.364524] Lustre: 26701:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 763.371043] Lustre: 26701:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.379835] Lustre: 26701:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.391698] Lustre: 26701:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.405644] Lustre: 26701:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.413931] Lustre: 26701:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.891096] Lustre: 26701:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 774.906161] Lustre: 26701:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 774.917260] Lustre: 26701:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 774.929826] Lustre: 26701:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.937165] Lustre: 26701:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 774.943340] Lustre: 26701:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.949550] Lustre: 26701:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 774.964572] Lustre: 26701:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.979656] Lustre: 26701:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 774.988093] Lustre: 26701:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.996730] Lustre: 26701:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 775.006756] Lustre: 26701:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 827.876552] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 827.878327] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.919781] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 832.996755] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 838.114883] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 841.184124] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 841.326565] Lustre: server umount lustre-OST07c6 complete [ 850.593796] Lustre: server umount lustre-MDT0000 complete [ 866.756163] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 870.158131] Key type lgssc unregistered [ 870.407117] LNet: 28746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.415144] LNetError: 28746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.436286] LNet: Removed LNI 192.168.203.122@tcp [ 871.229490] Key type .llcrypt unregistered [ 871.231713] Key type ._llcrypt unregistered [ 897.636183] Key type ._llcrypt registered [ 897.638624] Key type .llcrypt registered [ 897.766903] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 899.345613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.379221] alg: No test for adler32 (adler32-zlib) [ 900.557388] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 900.860078] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 902.568173] Key type lgssc registered [ 904.271584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 914.541382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 914.588429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.971358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.674258] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 926.186831] LustreError: 30023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.217831] LustreError: 30023:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 930.205186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.615219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.168365] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 942.526796] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 949.075290] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 956.277351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 956.742989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.841143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 963.253871] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 971.432639] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 991.713308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.716797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.725807] LustreError: Skipped 1 previous similar message [ 991.728425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.762691] Lustre: Skipped 1 previous similar message [ 995.463335] Lustre: server umount lustre-OST0000 complete [ 1001.955068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.966297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1001.974256] Lustre: Skipped 1 previous similar message [ 1001.986296] Lustre: Skipped 2 previous similar messages [ 1005.891259] Lustre: server umount lustre-MDT0000 complete [ 1007.077084] LustreError: 30023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.107854] LustreError: 30023:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1009.655670] LustreError: 30002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837277 with bad export cookie 11895725767541249333 [ 1009.673639] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.939421] Lustre: server umount lustre-MDT0001 complete [ 1021.676291] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1025.273528] Key type lgssc unregistered [ 1025.542718] LNet: 33762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.554677] LNetError: 33762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.599856] LNet: Removed LNI 192.168.203.122@tcp [ 1027.713425] Key type .llcrypt unregistered [ 1027.716085] Key type ._llcrypt unregistered [ 1048.564853] Key type ._llcrypt registered [ 1048.570353] Key type .llcrypt registered [ 1048.693893] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1049.733788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1049.765660] alg: No test for adler32 (adler32-zlib) [ 1050.860659] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 1051.109249] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1052.768230] Key type lgssc registered [ 1053.904108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.025792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.080846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.706427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.398295] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1073.123263] LustreError: 35035:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.153213] LustreError: 35035:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1075.594794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.989401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.882515] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1087.315832] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1094.490448] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1105.080581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.422690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.506569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1111.936193] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1119.891643] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1133.027454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1133.044256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.063564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1136.108550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1136.108945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.109220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.410882] Lustre: server umount lustre-OST0000 complete [ 1142.754202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1142.761783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.778455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.790262] Lustre: Skipped 1 previous similar message [ 1146.338379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.350916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1146.368485] Lustre: Skipped 1 previous similar message [ 1147.862273] Lustre: server umount lustre-MDT0000 complete [ 1151.462947] LustreError: 35035:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.484248] LustreError: 35035:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1151.819890] LustreError: 35016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837419 with bad export cookie 117405942668154631 [ 1151.837910] LustreError: 35016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1151.842191] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.237848] Lustre: server umount lustre-MDT0001 complete [ 1162.824948] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1165.444955] Key type lgssc unregistered [ 1165.798832] LNet: 38569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.802794] LNetError: 38569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.826699] LNet: Removed LNI 192.168.203.122@tcp [ 1166.730753] Key type .llcrypt unregistered [ 1166.741782] Key type ._llcrypt unregistered [ 1191.490940] Key type ._llcrypt registered [ 1191.493653] Key type .llcrypt registered [ 1191.629414] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1192.745324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1192.968638] alg: No test for adler32 (adler32-zlib) [ 1194.086368] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 1194.403612] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1196.064481] Key type lgssc registered [ 1197.181699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1204.320719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1204.363538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.755145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.910515] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1215.984049] LustreError: 39995:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1216.014164] LustreError: 39995:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1217.628535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.970352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.022241] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1231.033698] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1237.369903] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1245.663952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.214724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.318930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1253.588844] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1261.080975] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1267.685752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.687634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.711056] Lustre: Skipped 2 previous similar messages [ 1267.717317] Lustre: Skipped 2 previous similar messages [ 1272.801274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.805953] Lustre: Skipped 2 previous similar messages [ 1273.330591] Lustre: server umount lustre-MDT0000 complete [ 1277.399093] LustreError: 39977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837545 with bad export cookie 14363200783837926922 [ 1277.424965] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.921696] LustreError: 40984:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.931957] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.966094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1277.978077] Lustre: Skipped 1 previous similar message [ 1283.041277] LustreError: 40984:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1283.045144] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1283.064374] LustreError: 40984:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1283.075822] Lustre: Skipped 1 previous similar message [ 1283.990346] Lustre: server umount lustre-MDT0001 complete [ 1305.853623] Lustre: server umount lustre-OST0000 complete [ 1321.459713] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1324.149739] Key type lgssc unregistered [ 1324.505775] LNet: 43863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.516368] LNetError: 43863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.534856] LNet: Removed LNI 192.168.203.122@tcp [ 1325.471765] Key type .llcrypt unregistered [ 1325.473468] Key type ._llcrypt unregistered [ 1349.336129] Key type ._llcrypt registered [ 1349.338734] Key type .llcrypt registered [ 1349.482125] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1350.513399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1350.559707] alg: No test for adler32 (adler32-zlib) [ 1351.633707] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 1351.900826] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1353.560165] Key type lgssc registered [ 1354.972882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1363.380669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1363.396804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.923890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.089309] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1375.205141] LustreError: 45281:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.234692] LustreError: 45281:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1379.416580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.818337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1384.219750] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1391.808439] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1397.384384] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1404.491268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1404.846519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.980164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1411.719461] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1419.954334] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1430.497229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1430.505601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.510267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1430.532136] Lustre: Skipped 1 previous similar message [ 1432.225298] Lustre: server umount lustre-OST0000 complete [ 1450.983711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.992780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.993555] Lustre: Skipped 1 previous similar message [ 1451.007583] Lustre: Skipped 2 previous similar messages [ 1456.101549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.115248] Lustre: Skipped 1 previous similar message [ 1461.218696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.228122] Lustre: Skipped 1 previous similar message [ 1464.288168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1464.449480] Lustre: server umount lustre-MDT0000 complete [ 1466.341924] LustreError: 46011:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.365995] LustreError: 46011:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1468.163479] LustreError: 45262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837735 with bad export cookie 1239465664433940775 [ 1468.175522] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1474.639814] Lustre: server umount lustre-MDT0001 complete [ 1484.699318] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1487.288614] Key type lgssc unregistered [ 1487.557535] LNet: 48999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.569604] LNetError: 48999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1487.593692] LNet: Removed LNI 192.168.203.122@tcp [ 1488.452123] Key type .llcrypt unregistered [ 1488.454913] Key type ._llcrypt unregistered [ 1511.149110] Key type ._llcrypt registered [ 1511.151769] Key type .llcrypt registered [ 1511.243868] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1511.935217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1511.943094] alg: No test for adler32 (adler32-zlib) [ 1513.161248] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 1513.497880] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1515.212507] Key type lgssc registered [ 1516.510765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1524.125589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1524.149206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1525.607925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.969079] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1535.979593] LustreError: 50414:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.992490] LustreError: 50414:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1537.538603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.811427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1542.149929] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1550.884374] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1556.802455] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1564.181557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1564.511223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.210137] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1570.684850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1576.792547] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1586.379605] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1590.111666] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1602.702384] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-27 9h37m47s [ 1604.397797] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1608.675202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.678154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.694920] Lustre: Skipped 2 previous similar messages [ 1608.724402] Lustre: Skipped 2 previous similar messages [ 1609.766879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.785099] Lustre: Skipped 2 previous similar messages [ 1611.154386] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1611.161805] Lustre: Skipped 1 previous similar message [ 1612.843895] Lustre: server umount lustre-MDT0000 complete [ 1615.845305] LustreError: 50414:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1615.873171] LustreError: 50414:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1616.908859] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1617.240747] LustreError: 50411:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.996910] LustreError: 50409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1622.006771] LustreError: 50409:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1624.091923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.244067] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1624.521306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.594950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1625.523529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1629.773921] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1633.174234] Lustre: DEBUG MARKER: Wait for df (36753) ... [ 1634.770320] Lustre: DEBUG MARKER: done [ 1639.393690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1639.404573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.428079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1639.912966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.678519] Lustre: server umount lustre-OST0000 complete [ 1651.172628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.189141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1651.198944] Lustre: Skipped 1 previous similar message [ 1651.200456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1651.200463] Lustre: Skipped 3 previous similar messages [ 1656.385822] Lustre: server umount lustre-MDT0000 complete [ 1658.337464] LustreError: 51451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1658.357387] LustreError: 51451:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1660.217378] LustreError: 50393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837927 with bad export cookie 11789548259723781087 [ 1660.220047] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1660.238452] LustreError: 50393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1660.607466] Lustre: server umount lustre-MDT0001 complete [ 1672.096436] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1675.108790] Key type lgssc unregistered [ 1675.434710] LNet: 55345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.441780] LNetError: 55345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.456171] LNet: Removed LNI 192.168.203.122@tcp [ 1676.496148] Key type .llcrypt unregistered [ 1676.498604] Key type ._llcrypt unregistered [ 1715.152348] Key type ._llcrypt registered [ 1715.157763] Key type .llcrypt registered [ 1715.287417] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1716.497490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.629386] alg: No test for adler32 (adler32-zlib) [ 1717.670142] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 1717.930444] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1719.664190] Key type lgssc registered [ 1720.815230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1729.010947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1729.030676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.373428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1730.406459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1730.588307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.392934] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1744.819261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.911548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1744.943777] Lustre: 57552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1745.222330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1749.942843] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1757.509666] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1762.801511] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1769.444322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1769.617840] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1769.810374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.545935] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1778.802965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1782.005063] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1789.174148] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-27 9h40m54s [ 1790.607885] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1798.298658] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1800.908523] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1800.964419] LustreError: 57561:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1821.047687] Lustre: lustre-MDT0000: Client 0fc556b6-1d9b-4168-9520-e28da3d707d6 (at 192.168.203.22@tcp) reconnecting [ 1841.017044] LustreError: 57561:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1844.224521] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1847.226367] Lustre: DEBUG MARKER: done [ 1855.458955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.471666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.493163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.972188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.981401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.010255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.781978] Lustre: server umount lustre-OST0000 complete [ 1866.210668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.235628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.243775] Lustre: Skipped 1 previous similar message [ 1870.305869] Lustre: server umount lustre-MDT0000 complete [ 1874.487887] LustreError: 57544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838142 with bad export cookie 12534977811392475215 [ 1874.488142] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1874.514593] LustreError: 57544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1875.019642] Lustre: server umount lustre-MDT0001 complete [ 1886.357031] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1889.306282] Key type lgssc unregistered [ 1889.682307] LNet: 61887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.699777] LNetError: 61887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.715192] LNet: Removed LNI 192.168.203.122@tcp [ 1890.518501] Key type .llcrypt unregistered [ 1890.522947] Key type ._llcrypt unregistered [ 1934.173356] Key type ._llcrypt registered [ 1934.177135] Key type .llcrypt registered [ 1934.316817] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1935.479500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1935.627552] alg: No test for adler32 (adler32-zlib) [ 1936.787280] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 1937.071922] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1938.784160] Key type lgssc registered [ 1940.064692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.042656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.065113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.386719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1945.413755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1945.592818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.597815] Lustre: Failing over lustre-MDT0000 [ 1946.874411] Lustre: server umount lustre-MDT0000 complete [ 1953.403996] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:43:38 (1787838218) [ 1960.030922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.458827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.808668] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1973.023161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.112866] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1973.161405] Lustre: 64686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1973.370858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1977.412205] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1986.988032] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1994.573669] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2003.380034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2003.666474] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2003.949563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.708753] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2011.276498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2018.384152] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2024.585195] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2034.679303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2034.687036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.708030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.230327] Lustre: server umount lustre-MDT0000 complete [ 2042.573207] LustreError: 64677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838310 with bad export cookie 9865427646477785473 [ 2042.591610] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2042.595692] LustreError: 64677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.982772] Lustre: server umount lustre-MDT0001 complete [ 2047.041936] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2052.247335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2061.793415] Lustre: 63610:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838313/real 1787838313] req@0000000099c31c06 x1874684232640512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2061.847431] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.874644] Lustre: Skipped 2 previous similar messages [ 2063.481447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.968147] Lustre: 63610:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838315/real 1787838315] req@00000000e7382bbf x1874684232640640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2066.913086] Lustre: 63612:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838318/real 1787838318] req@00000000341227c8 x1874684232640768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2068.077058] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x88e90a2cd9a06264 [ 2068.091098] Lustre: MGC192.168.203.122@tcp: Connection restored to 0@lo (at 0@lo) [ 2068.571823] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.587804] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2068.719114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.888734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2069.664989] Lustre: 63612:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838320/real 1787838320] req@00000000aa3568bf x1874684232641024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2071.712996] LustreError: 66537:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.748164] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.769970] LustreError: 66536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2073.761310] Lustre: 63610:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838325/real 1787838325] req@00000000b8c3a911 x1874684232641152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2074.097446] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2078.185572] LustreError: 68457:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.245328] LustreError: 68457:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2083.326555] LustreError: 68459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.350282] LustreError: 68459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2086.182440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.658297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.740497] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2091.941699] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2102.228561] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2108.988895] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2117.603952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2117.619454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.640220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.653527] Lustre: Skipped 3 previous similar messages [ 2121.698988] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.702931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2121.720594] Lustre: Skipped 2 previous similar messages [ 2123.751978] Lustre: server umount lustre-MDT0000 complete [ 2126.823736] LustreError: 69195:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.836215] LustreError: 69195:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2128.499231] LustreError: 64677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838396 with bad export cookie 9865427646477787748 [ 2128.502745] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.507541] LustreError: 64677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2128.812905] Lustre: server umount lustre-MDT0001 complete [ 2136.300222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2147.296500] Lustre: 63611:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838399/real 1787838399] req@00000000132b176f x1874684232671232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2147.314865] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.327499] Lustre: Skipped 1 previous similar message [ 2147.537699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.764834] LustreError: 66537:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.833048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.915310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2158.764431] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2168.750060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.673808] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2174.453225] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2174.458959] Lustre: Skipped 1 previous similar message [ 2176.553400] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2181.361735] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2187.702379] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2194.915385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2194.923882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.947518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.697380] Lustre: server umount lustre-MDT0000 complete [ 2202.081626] LustreError: 71143:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.102147] LustreError: 71143:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 2204.500855] LustreError: 67600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838472 with bad export cookie 9865427646477789127 [ 2204.501840] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.520787] LustreError: 67600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2204.930731] Lustre: server umount lustre-MDT0001 complete [ 2211.662673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2215.121498] Lustre: DEBUG MARKER: files compared the same [ 2227.564855] Lustre: server umount lustre-OST0000 complete [ 2245.781873] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2248.712502] Key type lgssc unregistered [ 2249.049899] LNet: 74424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.060498] LNetError: 74424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.090813] LNet: Removed LNI 192.168.203.122@tcp [ 2251.078549] Key type .llcrypt unregistered [ 2251.084694] Key type ._llcrypt unregistered [ 2279.117420] Key type ._llcrypt registered [ 2279.120394] Key type .llcrypt registered [ 2279.224863] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2280.731145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.795237] alg: No test for adler32 (adler32-zlib) [ 2282.284795] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 2282.592674] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2284.328236] Key type lgssc registered [ 2285.136614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2294.274621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2294.294632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.642295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2300.337935] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 2306.020170] LustreError: 75843:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.050202] LustreError: 75843:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2310.142906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.475305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2315.180886] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 2322.792820] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2329.356409] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2336.924158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.400877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.476671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2343.951346] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 2351.722310] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2366.433394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.442460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.456594] Lustre: Skipped 1 previous similar message [ 2371.559253] LustreError: 77669:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.683474] Lustre: server umount lustre-OST0000 complete [ 2376.684348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.699648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.703153] Lustre: Skipped 2 previous similar messages [ 2381.793789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2381.803038] Lustre: Skipped 1 previous similar message [ 2381.979454] Lustre: server umount lustre-MDT0000 complete [ 2385.711757] LustreError: 75822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838653 with bad export cookie 15106177280464666985 [ 2385.718252] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2385.723789] LustreError: 75822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2386.224366] Lustre: server umount lustre-MDT0001 complete [ 2397.052259] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2399.846104] Key type lgssc unregistered [ 2400.209508] LNet: 79399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.217241] LNetError: 79399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.243508] LNet: Removed LNI 192.168.203.122@tcp [ 2401.047183] Key type .llcrypt unregistered [ 2401.050286] Key type ._llcrypt unregistered [ 2425.308511] Key type ._llcrypt registered [ 2425.324249] Key type .llcrypt registered [ 2425.459249] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2427.002402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2427.039372] alg: No test for adler32 (adler32-zlib) [ 2428.268759] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 2428.689928] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2430.411772] Key type lgssc registered [ 2432.075560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2440.741784] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2440.779482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2456.416570] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2461.536454] LustreError: 80791:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 2488.288924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.894668] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2505.216185] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2512.135658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.247688] Lustre: *** cfs_fail_loc=706, val=0*** [ 2512.650350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.723842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2517.072822] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2517.994697] LustreError: 80816:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.026579] LustreError: 80816:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2523.129605] LustreError: 80817:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.563817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.065358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2530.061294] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2537.230617] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2543.014198] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2546.661218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.666198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.668087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2546.701558] Lustre: Skipped 1 previous similar message [ 2551.531658] Lustre: server umount lustre-OST0000 complete [ 2556.897413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.911248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2556.912242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.912248] Lustre: Skipped 1 previous similar message [ 2556.917929] Lustre: Skipped 1 previous similar message [ 2562.019296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.031560] Lustre: Skipped 2 previous similar messages [ 2562.961106] Lustre: server umount lustre-MDT0000 complete [ 2567.142529] LustreError: 81881:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.164356] LustreError: 81881:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2567.611274] LustreError: 80798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838835 with bad export cookie 2725696116516807283 [ 2567.616050] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2567.622046] LustreError: 80798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2568.182234] Lustre: server umount lustre-MDT0001 complete [ 2578.906540] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2581.814791] Key type lgssc unregistered [ 2582.118468] LNet: 84412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2582.128415] LNetError: 84412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2582.156778] LNet: Removed LNI 192.168.203.122@tcp [ 2583.171911] Key type .llcrypt unregistered [ 2583.174512] Key type ._llcrypt unregistered [ 2613.861722] Key type ._llcrypt registered [ 2613.863890] Key type .llcrypt registered [ 2613.984356] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2615.288356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2615.330089] alg: No test for adler32 (adler32-zlib) [ 2616.491961] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 2616.760543] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2618.490061] Key type lgssc registered [ 2619.900219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2627.648911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2627.671838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.721214] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2641.432585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.045848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2645.828817] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2647.521283] LustreError: 86517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.543082] LustreError: 86517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2651.621498] LustreError: 86521:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.640543] LustreError: 86521:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2652.896031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.152930] LustreError: 86520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.190975] LustreError: 86520:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2653.196920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.275423] LustreError: 86521:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.288810] LustreError: 86521:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2658.828411] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2663.405380] LustreError: 86516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.432453] LustreError: 86516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2666.638485] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.664378] Lustre: MGS: Client 5cb44c75-d859-4b3b-accd-342177207a7a (at 0@lo) reconnecting [ 2666.698834] LustreError: 86531:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2666.939360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.029546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2671.089738] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2687.473683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2687.480884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.519119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.859796] Lustre: server umount lustre-OST0000 complete [ 2697.704689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.710598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2697.732159] Lustre: Skipped 1 previous similar message [ 2697.744335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.771532] Lustre: Skipped 2 previous similar messages [ 2701.159185] Lustre: server umount lustre-MDT0000 complete [ 2702.817704] LustreError: 86516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.832640] LustreError: 86516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2705.132623] Lustre: server umount lustre-MDT0001 complete [ 2709.326474] Lustre: server umount MGS complete [ 2720.432290] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:56:25 (1787838985) [ 2749.945246] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 2758.718164] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2798.097117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.741194] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2813.523684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.697309] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2813.733390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2813.832880] Lustre: lustre-MDT0001: new disk, initializing [ 2813.967918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2814.008509] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2814.017061] Lustre: 93682:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2814.034889] LustreError: 93682:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2818.644766] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2829.138516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2829.450523] Lustre: lustre-OST0000: new disk, initializing [ 2829.457873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2829.470832] Lustre: 94450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2829.530385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.787317] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2836.578810] Lustre: 94774:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ce85c1e2 x1874684945668736/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787839120 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2844.948513] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2851.491045] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2851.511373] Lustre: MGS: Client e60d63d5-6b20-4944-96a2-5b85ddc899ac (at 0@lo) reconnecting [ 2851.554871] Lustre: 92952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2851.577678] LustreError: 92952:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2851.585072] LustreError: 92952:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2851.607219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2851.712068] Lustre: lustre-MDT0000: new disk, initializing [ 2851.831554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.852183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2851.867266] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2852.903856] Lustre: 95519:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000001620746 x1874684945675520/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787839136 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2855.847788] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2858.017051] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2861.993252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2862.007931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2862.078694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2887.143272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.145877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2887.160964] Lustre: Skipped 2 previous similar messages [ 2887.170680] Lustre: Skipped 1 previous similar message [ 2891.655632] Lustre: server umount lustre-OST0000 complete [ 2897.382160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.390453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2897.393343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2897.406135] Lustre: Skipped 1 previous similar message [ 2897.454534] Lustre: Skipped 1 previous similar message [ 2902.503421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2902.518286] Lustre: Skipped 1 previous similar message [ 2910.176117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2910.256676] Lustre: server umount lustre-MDT0000 complete [ 2911.714454] LustreError: 93652:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.751279] LustreError: 93652:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2919.895332] Lustre: server umount lustre-MDT0001 complete [ 2923.021555] Lustre: server umount MGS complete [ 2933.488213] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:59:58 (1787839198) [ 2949.050178] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2951.316970] Key type lgssc unregistered [ 2951.592675] LNet: 98021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2951.607192] LNetError: 98021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2951.626503] LNet: Removed LNI 192.168.203.122@tcp [ 2952.506350] Key type .llcrypt unregistered [ 2952.508352] Key type ._llcrypt unregistered [ 2968.136302] Key type ._llcrypt registered [ 2968.138262] Key type .llcrypt registered [ 2968.228915] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2969.393842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2969.431096] alg: No test for adler32 (adler32-zlib) [ 2970.468530] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 2970.747741] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2972.417622] Key type lgssc registered [ 2973.566035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2981.359723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2981.384154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.817682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.569825] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2993.125809] LustreError: 99241:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.141237] LustreError: 99241:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2995.583271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.913303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3000.492619] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3007.717717] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3013.556271] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3020.931130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.314333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.466074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3027.888966] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3035.805546] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3047.914882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3047.930993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.961116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3050.977930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3050.982779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3050.995407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3051.017125] Lustre: Skipped 1 previous similar message [ 3053.956353] Lustre: server umount lustre-OST0000 complete [ 3061.217750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.251850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3061.256626] Lustre: Skipped 1 previous similar message [ 3064.412920] Lustre: server umount lustre-MDT0000 complete [ 3066.340621] LustreError: 99241:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.359187] LustreError: 99241:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3067.856449] LustreError: 99223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839335 with bad export cookie 9916838414214427026 [ 3067.862389] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.870248] LustreError: 99223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3068.315381] Lustre: server umount lustre-MDT0001 complete [ 3080.092330] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3082.582759] Key type lgssc unregistered [ 3082.880773] LNet: 102790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3082.897375] LNetError: 102790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3082.911421] LNet: Removed LNI 192.168.203.122@tcp [ 3083.704890] Key type .llcrypt unregistered [ 3083.707542] Key type ._llcrypt unregistered [ 3098.554941] Key type ._llcrypt registered [ 3098.559643] Key type .llcrypt registered [ 3098.701355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3099.441344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3099.460491] alg: No test for adler32 (adler32-zlib) [ 3100.592650] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 3100.883673] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3102.569234] Key type lgssc registered [ 3103.722292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3109.281330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3109.287360] LustreError: 103876:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3109.307765] LustreError: 103875:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3109.313259] LustreError: 103876:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 3109.318498] LustreError: 103876:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3109.330347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.332972] LustreError: 103876:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3110.799995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.056904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.404985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3122.896175] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3127.993833] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing lsmod [ 3130.657584] LustreError: 105076:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3130.705236] LustreError: 105083:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3130.710974] LustreError: 105076:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4956 [ 3130.715961] LustreError: 105076:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3130.730696] LustreError: 105076:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3130.736308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3131.021577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.925530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3133.934378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.943986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3136.486971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3136.490812] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.497113] Lustre: Skipped 1 previous similar message [ 3136.519562] Lustre: Skipped 1 previous similar message [ 3139.700314] Lustre: server umount lustre-MDT0000 complete [ 3141.604466] LustreError: 103913:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.622611] LustreError: 103913:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3143.067256] LustreError: 103889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839410 with bad export cookie 12987183388707183647 [ 3143.068601] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3143.081094] LustreError: 103889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3143.103651] Lustre: 103905:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3143.378898] Lustre: server umount lustre-MDT0001 complete [ 3157.119481] Lustre: server umount lustre-OST0000 complete [ 3165.274200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.825082] LustreError: 106145:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.961583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.788432] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3171.313094] LustreError: 106146:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.406178] LustreError: 106145:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.520706] LustreError: 106146:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.497298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.155256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3190.433269] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3198.842285] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3204.151587] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3211.868431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3212.271820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.951636] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3226.027380] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3235.833624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3243.640104] Lustre: server umount lustre-OST0000 complete [ 3250.663577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3250.674945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.686625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3250.697094] Lustre: Skipped 1 previous similar message [ 3256.290450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3256.300377] Lustre: Skipped 3 previous similar messages [ 3257.406726] Lustre: server umount lustre-MDT0000 complete [ 3259.362087] Lustre: 103276:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839511/real 1787839511] req@000000007d08ee53 x1874685453189632/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787839527 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3259.402773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.422210] Lustre: Skipped 1 previous similar message [ 3259.435270] LustreError: 106145:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.456742] LustreError: 106145:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3261.428838] LustreError: 106125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839529 with bad export cookie 12987183388707184319 [ 3261.435584] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3261.446864] LustreError: 106125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3261.723968] Lustre: server umount lustre-MDT0001 complete [ 3273.579896] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3277.072735] Key type lgssc unregistered [ 3277.356118] LNet: 109701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3277.374262] LNetError: 109701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3278.446724] LNet: Removed LNI 192.168.203.122@tcp [ 3279.266240] Key type .llcrypt unregistered [ 3279.268819] Key type ._llcrypt unregistered [ 3303.217822] Key type ._llcrypt registered [ 3303.223091] Key type .llcrypt registered [ 3303.317982] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3304.524978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3304.545426] alg: No test for adler32 (adler32-zlib) [ 3305.647104] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 3306.047852] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3307.760221] Key type lgssc registered [ 3308.875807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3315.822620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3315.867146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.422139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.400572] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3327.976516] LustreError: 111121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3328.018859] LustreError: 111121:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3329.093563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.425665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3332.853411] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3339.572882] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3344.955647] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3351.277372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3351.610141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.036885] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3362.940676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3365.228417] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3369.747358] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3373.052280] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3378.661103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.661746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3378.664546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3378.676080] Lustre: Skipped 1 previous similar message [ 3383.536855] Lustre: server umount lustre-OST0000 complete [ 3388.899235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.919558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3388.928966] Lustre: Skipped 2 previous similar messages [ 3393.319624] Lustre: server umount lustre-MDT0000 complete [ 3394.020423] LustreError: 111122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.036546] LustreError: 111122:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3396.690488] LustreError: 114005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839664 with bad export cookie 213112853370636067 [ 3396.692762] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3396.721691] LustreError: 114005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3397.118448] Lustre: server umount lustre-MDT0001 complete [ 3406.407841] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3408.801663] Key type lgssc unregistered [ 3409.071494] LNet: 114758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3409.080697] LNetError: 114758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3409.092763] LNet: Removed LNI 192.168.203.122@tcp [ 3409.920922] Key type .llcrypt unregistered [ 3409.923844] Key type ._llcrypt unregistered [ 3426.732043] Key type ._llcrypt registered [ 3426.734305] Key type .llcrypt registered [ 3426.892408] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3427.591324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3427.664137] alg: No test for adler32 (adler32-zlib) [ 3428.715511] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 3428.894445] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3430.544170] Key type lgssc registered [ 3431.585904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3437.756285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3437.779159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.207185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3443.766107] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3448.811853] LustreError: 115981:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.850604] LustreError: 115981:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3452.804441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.363565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3458.579552] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3467.047234] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3472.857178] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3479.121050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3479.453666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3483.565957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3486.694898] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3494.978455] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3509.220693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3509.233899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.258722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3513.318107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3513.321962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3513.334495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.631290] Lustre: server umount lustre-OST0000 complete [ 3518.434742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.436347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.456275] Lustre: Skipped 1 previous similar message [ 3518.478577] Lustre: Skipped 2 previous similar messages [ 3523.555968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3523.560139] Lustre: Skipped 1 previous similar message [ 3524.318473] Lustre: server umount lustre-MDT0000 complete [ 3527.146177] LustreError: 115959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839794 with bad export cookie 17672427221231666807 [ 3527.148704] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3527.168039] LustreError: 115959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3527.390981] Lustre: server umount lustre-MDT0001 complete [ 3536.089719] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3538.431578] Key type lgssc unregistered [ 3538.698768] LNet: 119537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3538.707975] LNetError: 119537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3538.720427] LNet: Removed LNI 192.168.203.122@tcp [ 3539.498265] Key type .llcrypt unregistered [ 3539.500180] Key type ._llcrypt unregistered [ 3560.586827] Key type ._llcrypt registered [ 3560.589575] Key type .llcrypt registered [ 3560.710547] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3561.732600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3561.763576] alg: No test for adler32 (adler32-zlib) [ 3562.856170] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 3563.084177] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3564.768196] Key type lgssc registered [ 3565.826946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3572.603219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3572.635443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.085511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3578.290350] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3583.465223] LustreError: 120933:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3583.479225] LustreError: 120933:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3586.189319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.548576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3590.484702] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3597.668551] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3603.345920] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3610.511746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3610.931540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.288746] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3619.143233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3625.648602] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3633.590335] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3638.231791] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3638.240440] Lustre: Skipped 2 previous similar messages [ 3643.006651] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3643.089910] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3643.287787] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3643.298598] Lustre: Skipped 2 previous similar messages [ 3650.646881] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3650.656040] Lustre: Skipped 1 previous similar message [ 3655.555588] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.22@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3655.565277] Lustre: Skipped 2 previous similar messages [ 3660.840967] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.22@tcp 192.168.0.[2,10]@tcp [ 3664.287628] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.22/24@tcp in log lustre-MDT0000 [ 3664.293777] Lustre: Skipped 2 previous similar messages [ 3666.708520] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.22/24@tcp [ 3666.712383] Lustre: Skipped 1 previous similar message [ 3678.689255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3678.695813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.705033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3679.204351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3679.208599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.216485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3680.024376] Lustre: server umount lustre-OST0000 complete [ 3683.812233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3683.816145] LustreError: Skipped 3 previous similar messages [ 3683.825654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.840385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.862688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.865879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.441258] Lustre: server umount lustre-MDT0000 complete [ 3691.000608] LustreError: 120934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.025365] LustreError: 120934:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3692.855664] LustreError: 120913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839960 with bad export cookie 13055327672095023103 [ 3692.865835] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3692.872663] LustreError: 120913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3693.263620] Lustre: server umount lustre-MDT0001 complete [ 3702.988508] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3705.485931] Key type lgssc unregistered [ 3705.719916] LNet: 125880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3705.725114] LNetError: 125880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3705.736587] LNet: Removed LNI 192.168.203.122@tcp [ 3706.521620] Key type .llcrypt unregistered [ 3706.524448] Key type ._llcrypt unregistered [ 3733.504329] Key type ._llcrypt registered [ 3733.509832] Key type .llcrypt registered [ 3733.637533] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3734.840421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3734.917913] alg: No test for adler32 (adler32-zlib) [ 3736.038214] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 3736.276488] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3737.986146] Key type lgssc registered [ 3738.900540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3746.509692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3746.527893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.054811] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3748.080489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.082848] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.22/24@tcp [ 3752.560243] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3757.548707] LustreError: 127441:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3757.576043] LustreError: 127441:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3761.560978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.954677] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3761.989412] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.22/24@tcp [ 3762.057450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3766.210865] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3773.031881] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3778.335891] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3784.808193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3785.198082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.789055] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3792.373264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3798.630766] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3807.713047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3807.715891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.721159] LustreError: Skipped 1 previous similar message [ 3807.722189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.745452] Lustre: Skipped 1 previous similar message [ 3812.841524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.850720] Lustre: Skipped 2 previous similar messages [ 3813.316924] Lustre: server umount lustre-OST0000 complete [ 3817.962857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.971995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3817.980799] Lustre: Skipped 1 previous similar message [ 3817.996860] Lustre: Skipped 1 previous similar message [ 3823.074139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.084404] Lustre: Skipped 1 previous similar message [ 3823.389309] Lustre: server umount lustre-MDT0000 complete [ 3827.563573] LustreError: 127423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840095 with bad export cookie 15268942600688106524 [ 3827.566563] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3827.569699] LustreError: 127423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3828.081308] Lustre: server umount lustre-MDT0001 complete [ 3839.286932] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3842.166979] Key type lgssc unregistered [ 3842.475609] LNet: 131033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3842.490679] LNetError: 131033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3843.569318] LNet: Removed LNI 192.168.203.122@tcp [ 3844.567883] Key type .llcrypt unregistered [ 3844.571445] Key type ._llcrypt unregistered [ 3874.939735] Key type ._llcrypt registered [ 3874.942371] Key type .llcrypt registered [ 3875.057792] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3890.578518] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3891.557795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3891.662087] alg: No test for adler32 (adler32-zlib) [ 3892.755982] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 3893.117729] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3894.840280] Key type lgssc registered [ 3896.172989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3928.683414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3928.698754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.102898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3930.173640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3930.274979] Lustre: lustre-MDT0000: new disk, initializing [ 3930.344785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.368088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3935.245366] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3947.212321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.362720] Lustre: 134432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3947.418562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3947.426324] Lustre: Skipped 1 previous similar message [ 3947.494638] Lustre: lustre-MDT0001: new disk, initializing [ 3947.594726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3947.623550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3947.636996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3952.568847] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3963.216892] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3969.913986] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3977.392279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.709250] Lustre: lustre-OST0000: new disk, initializing [ 3977.717430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3977.727386] Lustre: 136403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3977.853468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.510639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3984.524375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3984.693442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3984.818563] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3994.315492] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3999.486833] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3999.817684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4005.675944] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4005.944654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4010.465432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.473862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.480108] LustreError: Skipped 1 previous similar message [ 4010.487341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4010.503471] Lustre: Skipped 1 previous similar message [ 4014.335387] Lustre: server umount lustre-OST0000 complete [ 4019.178839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4019.195077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.208738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4019.216271] Lustre: Skipped 1 previous similar message [ 4020.707292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.707968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.554604] Lustre: server umount lustre-MDT0000 complete [ 4028.233840] LustreError: 135229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840295 with bad export cookie 2150212934135026652 [ 4028.240846] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4028.250253] LustreError: 135229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4028.814122] Lustre: server umount lustre-MDT0001 complete [ 4037.384753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.004903] LustreError: 138618:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.032209] LustreError: 138618:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4038.099384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.134483] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4043.235088] LustreError: 138619:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4047.335512] LustreError: 138618:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.714964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.109693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4057.697472] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4065.805520] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4070.992392] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4077.218945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4077.590621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4084.025061] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4091.754659] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4096.803894] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4097.028260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4102.247644] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4102.499396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4111.519047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4111.683626] Lustre: lustre-OST0001: new disk, initializing [ 4111.686895] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4111.692765] Lustre: 141988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4111.784474] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4118.141228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4118.154237] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4118.253896] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4118.691779] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4127.201309] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4127.409826] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4131.612322] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4131.883118] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4137.463719] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b0090022000.ost_server_uuid 50 [ 4138.930030] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b0090022000.ost_server_uuid in IDLE state after 0 sec [ 4149.219885] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4149.232812] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.241399] LustreError: 140442:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.247438] Lustre: Skipped 1 previous similar message [ 4149.391235] Lustre: server umount lustre-OST0001 complete [ 4154.343293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.353617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4154.354462] LustreError: 140440:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.354472] LustreError: 140440:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4154.354860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4154.354864] Lustre: Skipped 1 previous similar message [ 4154.374343] Lustre: Skipped 1 previous similar message [ 4154.383126] LustreError: Skipped 1 previous similar message [ 4159.461698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4159.478990] Lustre: Skipped 2 previous similar messages [ 4160.306446] Lustre: server umount lustre-OST0000 complete [ 4164.580243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4164.581701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4164.611762] Lustre: Skipped 1 previous similar message [ 4169.698764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4169.709731] Lustre: Skipped 3 previous similar messages [ 4178.400131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4178.664965] Lustre: server umount lustre-MDT0000 complete [ 4179.939256] LustreError: 138618:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4179.962043] LustreError: 138618:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4183.355383] LustreError: 138598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840451 with bad export cookie 2150212934135027534 [ 4183.393406] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4183.815353] Lustre: server umount lustre-MDT0001 complete [ 4208.518975] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4212.567679] Key type lgssc unregistered [ 4212.969764] LNet: 145001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4212.980072] LNetError: 145001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4213.004561] LNet: Removed LNI 192.168.203.122@tcp [ 4214.118078] Key type .llcrypt unregistered [ 4214.129605] Key type ._llcrypt unregistered [ 4254.424148] Key type ._llcrypt registered [ 4254.425727] Key type .llcrypt registered [ 4254.514806] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4268.048299] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4269.240637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4269.312962] alg: No test for adler32 (adler32-zlib) [ 4270.433536] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 4270.703888] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4272.344190] Key type lgssc registered [ 4273.564612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4304.383575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4304.425826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4305.572806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4305.602470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4305.664344] Lustre: lustre-MDT0000: new disk, initializing [ 4305.735978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.751253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4310.205722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4321.710715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.865024] Lustre: 149146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4321.957971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4321.963845] Lustre: Skipped 1 previous similar message [ 4322.077757] Lustre: lustre-MDT0001: new disk, initializing [ 4322.250139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4322.294669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4322.318442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4327.068071] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4337.234844] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4343.483906] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4351.091440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4351.446985] Lustre: lustre-OST0000: new disk, initializing [ 4351.450887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4351.457800] Lustre: 151126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4351.598542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4358.339714] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4360.899952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4360.911501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4361.051424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4369.123633] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4373.271061] Lustre: Failing over lustre-OST0000 [ 4375.531699] Lustre: server umount lustre-OST0000 complete [ 4391.904313] Lustre: 147374:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840643/real 1787840643] req@000000006fe4706e x1874686680008448/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787840659 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4391.917495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4392.928483] Lustre: 147376:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840644/real 1787840644] req@0000000069886cfb x1874686680009344/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787840660 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4392.934078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4392.953254] Lustre: 147376:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4395.256362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4395.705668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4395.726313] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4397.741912] Lustre: 147373:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840649/real 1787840649] req@00000000998fedab x1874686680010496/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787840665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4397.772946] Lustre: 147373:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4401.833081] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4402.073412] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4402.848213] Lustre: 147373:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840654/real 1787840654] req@00000000b0139452 x1874686680011520/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787840670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4402.955427] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4402.958837] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4405.798297] Lustre: Failing over lustre-MDT0000 [ 4406.070137] Lustre: server umount lustre-MDT0000 complete [ 4406.958654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.985471] Lustre: Skipped 1 previous similar message [ 4407.970187] Lustre: 147373:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840659/real 1787840659] req@000000005406b172 x1874686680012672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787840675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4408.027162] Lustre: 147373:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4412.389180] LustreError: 149153:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.417588] LustreError: 149153:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4417.505177] LustreError: 149159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.531945] LustreError: 149159:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4422.628719] Lustre: 147373:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840674/real 1787840674] req@00000000b0139452 x1874686680024064/t0(0) o400->MGC192.168.203.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787840690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4422.655222] LustreError: 150320:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.697259] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4422.765307] LustreError: 150320:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4427.287938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.749951] LustreError: 152381:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.770863] LustreError: 152381:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4432.864665] LustreError: 147372:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b143833b x1874686680028928/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4432.883886] LustreError: 150320:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.910098] LustreError: 150320:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4433.099908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4433.186053] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4434.186978] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4436.994439] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4438.504380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4438.519919] Lustre: Skipped 1 previous similar message [ 4438.560436] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4438.618517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4444.132388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4444.134560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4444.149751] LustreError: Skipped 1 previous similar message [ 4444.170518] Lustre: Skipped 1 previous similar message [ 4444.175913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.181968] Lustre: Skipped 1 previous similar message [ 4448.751686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4448.757224] Lustre: Skipped 1 previous similar message [ 4450.205347] Lustre: server umount lustre-OST0000 complete [ 4453.858148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4453.859986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.870288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4453.890875] Lustre: Skipped 2 previous similar messages [ 4458.982151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.993713] Lustre: Skipped 2 previous similar messages [ 4459.773653] Lustre: server umount lustre-MDT0000 complete [ 4463.316235] LustreError: 149141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840731 with bad export cookie 5833414790516911930 [ 4463.317642] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4463.323622] LustreError: 149141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4463.651099] Lustre: server umount lustre-MDT0001 complete [ 4474.268478] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4477.056861] Key type lgssc unregistered [ 4477.338704] LNet: 154789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4477.347089] LNetError: 154789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4477.373753] LNet: Removed LNI 192.168.203.122@tcp [ 4478.409086] Key type .llcrypt unregistered [ 4478.412127] Key type ._llcrypt unregistered [ 4513.798408] Key type ._llcrypt registered [ 4513.807439] Key type .llcrypt registered [ 4513.939173] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4515.844545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4515.916331] alg: No test for adler32 (adler32-zlib) [ 4517.132956] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 4517.551860] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4519.360911] Key type lgssc registered [ 4520.790393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4530.584762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4530.596544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.315530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4538.203242] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4541.412843] LustreError: 156311:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.461460] LustreError: 156311:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4546.537781] LustreError: 156310:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4550.257995] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4551.650509] LustreError: 156311:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4553.954319] LustreError: 156339:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4553.974894] LustreError: 156339:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4553.990217] LustreError: 156339:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4554.336987] Lustre: server umount lustre-MDT0000 complete [ 4569.603873] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4580.874945] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4617.467697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.744733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4617.764418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4617.816820] Lustre: lustre-MDT0000: new disk, initializing [ 4617.871489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.885560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4622.774266] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4633.466586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4633.610714] Lustre: 159643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4633.700941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4633.702958] Lustre: Skipped 1 previous similar message [ 4633.834443] Lustre: lustre-MDT0001: new disk, initializing [ 4634.060791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4634.159953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4634.175925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4640.670666] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4651.229250] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4657.677753] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4664.872053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4665.159903] Lustre: lustre-OST0000: new disk, initializing [ 4665.163785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4665.168933] Lustre: 161614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4665.264575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.880473] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4672.083078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4672.094208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4672.255925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4681.606364] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4686.515290] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4688.348511] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:29:13 (1787840953) [ 4915.681392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4915.687684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4915.704576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4916.192779] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.192779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.226774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4916.230636] Lustre: Skipped 2 previous similar messages [ 4920.600222] Lustre: server umount lustre-MDT0000 complete [ 4921.315153] LustreError: 159650:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.328703] LustreError: 159650:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4923.871989] LustreError: 159636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841191 with bad export cookie 17607225798524096382 [ 4923.878769] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4923.882863] LustreError: 159636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4926.432769] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.433052] LustreError: 159651:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.455485] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4928.480885] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4933.601035] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4938.208290] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4938.471890] Lustre: server umount lustre-MDT0001 complete [ 4948.692524] Lustre: server umount lustre-OST0000 complete [ 4958.810486] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4970.726233] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5006.394951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.648478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5006.682259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5006.771925] Lustre: lustre-MDT0000: new disk, initializing [ 5006.861109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5006.876211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5011.297346] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5021.783077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.873760] Lustre: 166273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5021.909481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5021.912072] Lustre: Skipped 1 previous similar message [ 5021.976160] Lustre: lustre-MDT0001: new disk, initializing [ 5022.060157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5022.093697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5022.108943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5026.860590] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5037.636256] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5043.247722] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5050.339387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5050.672085] Lustre: lustre-OST0000: new disk, initializing [ 5050.675383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5050.686958] Lustre: 168244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5050.812906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.023175] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5059.727281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5059.743921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5059.858579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5069.686318] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5074.988854] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5075.412663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5081.953744] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5082.309987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5085.675521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.678619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5085.691955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.701886] Lustre: Skipped 1 previous similar message [ 5091.207584] Lustre: server umount lustre-OST0000 complete [ 5095.905857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.924350] Lustre: Skipped 1 previous similar message [ 5101.725654] Lustre: server umount lustre-MDT0000 complete [ 5105.029740] LustreError: 168243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841372 with bad export cookie 17607225798524255212 [ 5105.040765] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5105.046323] LustreError: 168243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5105.390754] Lustre: server umount lustre-MDT0001 complete [ 5113.762481] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:36:19 (1787841379) [ 5136.137304] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5147.764096] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5183.462453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.709754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5183.735276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5183.813478] Lustre: lustre-MDT0000: new disk, initializing [ 5183.869249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.891598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5188.059954] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5199.006986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.143888] Lustre: 173004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5199.454568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5204.168952] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5214.661322] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5220.067197] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5226.921805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5227.189529] Lustre: lustre-OST0000: new disk, initializing [ 5227.192949] Lustre: Skipped 1 previous similar message [ 5227.196928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5227.204940] Lustre: Skipped 2 previous similar messages [ 5227.211580] Lustre: 174980:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5233.387704] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5233.723237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5233.734574] Lustre: Skipped 1 previous similar message [ 5233.740389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5233.816577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5242.484069] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5254.625565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5254.626143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5254.633571] LustreError: Skipped 2 previous similar messages [ 5254.636808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5254.661983] Lustre: Skipped 1 previous similar message [ 5254.675032] Lustre: Skipped 8 previous similar messages [ 5256.461459] Lustre: server umount lustre-OST0000 complete [ 5260.778320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5266.337723] Lustre: server umount lustre-MDT0000 complete [ 5269.988304] LustreError: 173014:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5270.016169] LustreError: 173014:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5270.080738] LustreError: 174979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841537 with bad export cookie 17607225798524256094 [ 5270.083685] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5270.093267] LustreError: 174979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5270.502548] Lustre: server umount lustre-MDT0001 complete [ 5277.518140] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:39:03 (1787841543) [ 5292.097647] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5300.874708] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5334.157544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.383642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5334.447703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5334.574600] Lustre: lustre-MDT0000: new disk, initializing [ 5334.705097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5334.721573] Lustre: Skipped 2 previous similar messages [ 5334.756656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5338.567524] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5348.152501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.320959] Lustre: 179301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5348.568936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5353.019604] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5362.145326] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5367.712937] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5374.497559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5374.805452] Lustre: 181278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5381.156799] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5383.756069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5383.854685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5389.918886] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5399.522627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5399.536100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.564902] Lustre: Skipped 2 previous similar messages [ 5399.570370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5399.576113] Lustre: Skipped 4 previous similar messages [ 5410.273510] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5410.469136] Lustre: server umount lustre-OST0000 complete [ 5420.675308] Lustre: server umount lustre-MDT0000 complete [ 5423.015921] LustreError: 179313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.040050] LustreError: 179313:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5424.557410] LustreError: 179292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841692 with bad export cookie 17607225798524257179 [ 5424.569282] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5424.575099] LustreError: 179292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5425.060186] Lustre: server umount lustre-MDT0001 complete [ 5436.634288] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5439.885691] Key type lgssc unregistered [ 5440.282890] LNet: 183140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5440.294643] LNetError: 183140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5440.308226] LNet: Removed LNI 192.168.203.122@tcp [ 5441.418419] Key type .llcrypt unregistered [ 5441.422816] Key type ._llcrypt unregistered [ 5465.985703] Key type ._llcrypt registered [ 5465.988294] Key type .llcrypt registered [ 5466.146166] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5467.164053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5467.194990] alg: No test for adler32 (adler32-zlib) [ 5468.321075] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 5468.710350] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5470.438053] Key type lgssc registered [ 5471.664305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5479.983961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5480.010241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.522400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5486.383573] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5491.682235] LustreError: 184557:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.702876] LustreError: 184557:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5495.801707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.117395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5500.554312] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5508.318493] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5514.408257] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5521.836301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5522.254291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5526.399577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5529.068470] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5537.214267] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5561.319024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.319823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5561.321056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5561.339924] Lustre: Skipped 1 previous similar message [ 5565.166719] Lustre: server umount lustre-OST0000 complete [ 5571.556202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.557856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5571.579342] Lustre: Skipped 1 previous similar message [ 5571.594707] Lustre: Skipped 2 previous similar messages [ 5575.787958] Lustre: server umount lustre-MDT0000 complete [ 5576.678104] LustreError: 184557:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.710948] LustreError: 184557:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5580.163302] LustreError: 184539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841847 with bad export cookie 17224477487127680485 [ 5580.163934] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5580.180449] LustreError: 184539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5580.616649] Lustre: server umount lustre-MDT0001 complete [ 5593.564495] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5596.959937] Key type lgssc unregistered [ 5597.356776] LNet: 188122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5597.375358] LNetError: 188122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5597.394838] LNet: Removed LNI 192.168.203.122@tcp [ 5598.393931] Key type .llcrypt unregistered [ 5598.396846] Key type ._llcrypt unregistered [ 5628.982339] Key type ._llcrypt registered [ 5628.984865] Key type .llcrypt registered [ 5629.154588] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5631.056521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5631.125590] alg: No test for adler32 (adler32-zlib) [ 5632.258541] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 5632.596819] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5634.327756] Key type lgssc registered [ 5635.528268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5644.396554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5644.425141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.779209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5651.277495] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5656.054458] LustreError: 189540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5656.087364] LustreError: 189540:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5657.083067] LustreError: 189541:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.194821] LustreError: 189540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5664.025744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5664.669471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5672.160410] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5681.467511] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5688.220910] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5695.849984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5696.325243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.156326] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5710.121760] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5716.787505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5721.952101] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 5722.083698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.094313] Lustre: Skipped 1 previous similar message [ 5726.891512] Lustre: server umount lustre-OST0000 complete [ 5734.057217] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b0090027800.ost_server_uuid 50 [ 5735.567757] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5751.781047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5751.791579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5751.793723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5751.811945] Lustre: Skipped 1 previous similar message [ 5751.814453] Lustre: Skipped 2 previous similar messages [ 5756.342261] Lustre: server umount lustre-MDT0000 complete [ 5756.901743] LustreError: 189541:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5756.931960] LustreError: 189541:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5760.633155] LustreError: 189520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842028 with bad export cookie 15396378351452074013 [ 5760.637062] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5760.648686] LustreError: 189520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5761.288170] Lustre: server umount lustre-MDT0001 complete [ 5770.893529] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:47:16 (1787842036) [ 5777.764780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.154901] LustreError: 193461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.230164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.187941] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5787.622792] LustreError: 193462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.661120] LustreError: 193462:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5792.015731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.410832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5797.586339] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5806.719506] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5815.283493] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5823.146762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.605297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.330735] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5838.064225] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5845.021652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5845.279709] Lustre: lustre-OST0001: new disk, initializing [ 5845.287389] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5845.297224] Lustre: 196299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5845.453574] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5847.557289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5851.274502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5851.297864] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5851.467741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5853.083449] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5863.396773] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5866.980098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5867.009369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5867.019048] Lustre: Skipped 1 previous similar message [ 5869.412353] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 5869.419099] Lustre: Skipped 1 previous similar message [ 5872.549392] Lustre: server umount lustre-OST0000 complete [ 5874.500259] LustreError: 195282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5878.759869] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5878.995350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5884.375537] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5884.638569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5902.820280] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5902.821660] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.835057] LustreError: Skipped 1 previous similar message [ 5902.837701] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5902.851040] Lustre: Skipped 2 previous similar messages [ 5902.871525] Lustre: Skipped 3 previous similar messages [ 5904.648119] Lustre: server umount lustre-OST0001 complete [ 5909.987034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5909.995258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.065338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5913.079844] Lustre: Skipped 2 previous similar messages [ 5914.723444] Lustre: server umount lustre-MDT0000 complete [ 5918.182368] LustreError: 193462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.214711] LustreError: 193462:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [ 5918.817346] LustreError: 193442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842186 with bad export cookie 15396378351452075084 [ 5918.826376] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5918.828021] LustreError: 193442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5919.230618] Lustre: server umount lustre-MDT0001 complete [ 5944.534265] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:50:09 (1787842209) [ 5951.510374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.862069] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5951.893205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5952.045368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.628182] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5964.721286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5964.788037] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5964.820426] Lustre: 199743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5969.547414] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5977.529362] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5983.236198] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5990.097420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5990.269385] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5996.555461] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5997.194246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6004.034336] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6011.166599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6011.210498] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6018.644357] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6026.714691] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6031.862566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6031.886678] Lustre: Skipped 1 previous similar message [ 6031.895407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6031.905041] Lustre: Skipped 1 previous similar message [ 6035.326380] Lustre: server umount lustre-OST0000 complete [ 6036.968177] LustreError: 201593:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.007442] LustreError: 201593:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6052.322263] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6052.325188] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.358934] Lustre: Skipped 1 previous similar message [ 6057.390791] Lustre: server umount lustre-OST0001 complete [ 6062.562838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6062.575611] LustreError: Skipped 1 previous similar message [ 6067.683173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6067.689255] Lustre: Skipped 6 previous similar messages [ 6075.361447] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6075.574653] Lustre: server umount lustre-MDT0000 complete [ 6079.014919] LustreError: 199735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842346 with bad export cookie 15396378351452076491 [ 6079.023632] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.027307] LustreError: 199735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6079.311319] Lustre: server umount lustre-MDT0001 complete [ 6102.009665] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:52:47 (1787842367) [ 6116.768107] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6126.724648] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6160.605961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.840693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6160.875917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6160.996365] Lustre: lustre-MDT0000: new disk, initializing [ 6161.085888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6161.101503] Lustre: Skipped 3 previous similar messages [ 6161.131838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6165.273929] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6175.137695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.311596] Lustre: 207838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6175.351134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6175.354910] Lustre: Skipped 1 previous similar message [ 6175.405630] Lustre: lustre-MDT0001: new disk, initializing [ 6175.528490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6175.542177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6179.941647] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6188.974375] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6194.625555] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6201.118521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6201.408467] Lustre: lustre-OST0000: new disk, initializing [ 6201.415774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6201.422867] Lustre: 209806:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6208.078747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6208.086774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6208.160771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6208.532924] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6219.917627] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6226.300412] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6226.669140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6232.099593] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6232.407554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6239.206451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6239.215427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.257722] Lustre: Skipped 2 previous similar messages [ 6239.289160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6239.301944] Lustre: Skipped 3 previous similar messages [ 6241.657551] Lustre: server umount lustre-OST0000 complete [ 6260.192856] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6260.495435] Lustre: server umount lustre-MDT0000 complete [ 6264.808840] LustreError: 207850:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6264.842281] LustreError: 207850:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 6264.937656] LustreError: 207831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842532 with bad export cookie 15396378351452078052 [ 6264.940442] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6264.958251] LustreError: 207831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6265.240609] Lustre: server umount lustre-MDT0001 complete [ 6273.956595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.034774] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6290.308819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6290.874055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6290.893097] Lustre: Skipped 3 previous similar messages [ 6296.719800] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6304.009562] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6309.167693] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6316.404790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6323.037295] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6330.340060] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6334.797374] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6334.997809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6339.561121] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6339.769170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6341.608670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6341.634714] Lustre: Skipped 4 previous similar messages [ 6343.636404] Lustre: server umount lustre-OST0000 complete [ 6350.548063] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6355.561311] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6385.301933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6392.316759] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6399.752581] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.495577] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6407.176703] Lustre: server umount lustre-OST0000 complete [ 6413.285674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6413.297300] Lustre: Skipped 10 previous similar messages [ 6417.478536] Lustre: server umount lustre-MDT0000 complete [ 6421.865139] LustreError: 216140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842689 with bad export cookie 15396378351452078934 [ 6421.877344] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.892814] LustreError: 216140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6422.387486] Lustre: server umount lustre-MDT0001 complete [ 6431.490461] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:58:17 (1787842697) [ 6438.157801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6443.458627] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6451.861065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6457.114550] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6465.994989] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6471.519570] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6478.093252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6484.296911] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6490.933622] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6495.007414] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6495.221464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6499.134110] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6499.356121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6505.564677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.742415] Lustre: lustre-OST0001: new disk, initializing [ 6505.746619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6505.751482] Lustre: 221360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6511.523149] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6514.281132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6514.298582] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6514.383801] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6520.138725] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6524.616506] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6524.800792] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6529.153650] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6529.376735] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6535.148863] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6535.153186] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.162271] LustreError: Skipped 3 previous similar messages [ 6535.184545] Lustre: Skipped 2 previous similar messages [ 6537.635333] Lustre: server umount lustre-OST0001 complete [ 6540.278846] LustreError: 221671:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.313635] LustreError: 221671:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 6544.014359] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6544.221096] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6548.046248] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6548.264265] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6579.192811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.400631] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6579.409564] Lustre: Skipped 6 previous similar messages [ 6584.816722] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6584.839734] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6584.856210] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6584.869807] Lustre: Skipped 1 previous similar message [ 6587.497846] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6595.428084] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6597.180388] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6600.161429] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6605.266866] Lustre: server umount lustre-OST0001 complete [ 6628.030152] Lustre: server umount lustre-OST0000 complete [ 6637.502643] Lustre: server umount lustre-MDT0000 complete [ 6640.915635] LustreError: 218728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842908 with bad export cookie 15396378351452079886 [ 6640.924515] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6640.936504] LustreError: 218728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6647.472246] Lustre: server umount lustre-MDT0001 complete [ 6670.489432] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:02:15 (1787842935) [ 6677.159427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.536346] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6677.575846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6683.226758] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6690.950240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.140605] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6691.218808] Lustre: 226613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6696.155107] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6703.313446] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6710.025318] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6717.799485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6717.983592] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6724.547075] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6731.427316] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6739.853279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6739.909058] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6744.953689] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6750.999243] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6752.441936] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6755.999908] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6756.182784] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6760.371583] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6760.627502] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6765.747573] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b0088160000.ost_server_uuid 50 [ 6766.980851] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b0088160000.ost_server_uuid in FULL state after 0 sec [ 6767.864612] Lustre: Permanently deactivating lustre-OST0001 [ 6767.885258] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6771.282227] Lustre: Permanently reactivating lustre-OST0001 [ 6778.338639] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6778.348228] LustreError: Skipped 4 previous similar messages [ 6778.363025] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6778.372727] Lustre: Skipped 10 previous similar messages [ 6780.874831] Lustre: server umount lustre-OST0001 complete [ 6798.304220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6798.517748] Lustre: server umount lustre-OST0000 complete [ 6804.449749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6804.464947] Lustre: Skipped 11 previous similar messages [ 6816.224132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6816.405369] Lustre: server umount lustre-MDT0000 complete [ 6819.601587] LustreError: 228479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843087 with bad export cookie 15396378351452081167 [ 6819.602845] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6819.613612] LustreError: 228479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6819.998490] Lustre: server umount lustre-MDT0001 complete [ 6841.776360] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:05:07 (1787843107) [ 6850.039239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.375423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6850.417266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6850.430258] Lustre: Skipped 5 previous similar messages [ 6855.413833] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6864.018301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.094060] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6864.128247] Lustre: 233311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6868.515839] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6875.916148] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6880.843852] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6887.611841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.898490] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6887.931153] Lustre: Permanently deactivating lustre-OST0000 [ 6887.946267] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6895.122029] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6895.194124] LustreError: 188850:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ac825b82 x1874688108590080/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6895.258606] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6903.634407] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6910.453780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6910.546517] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6916.109794] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6920.800466] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6921.702973] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6922.923331] Lustre: Permanently reactivating lustre-OST0000 [ 6922.935387] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6922.940463] Lustre: Skipped 2 previous similar messages [ 6930.504966] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6930.513577] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6930.527066] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6930.568464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6938.807288] Lustre: server umount lustre-OST0001 complete [ 6941.157876] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6941.168013] LustreError: Skipped 2 previous similar messages [ 6948.894769] Lustre: server umount lustre-OST0000 complete [ 6958.991340] Lustre: server umount lustre-MDT0000 complete [ 6962.994553] LustreError: 236903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843230 with bad export cookie 15396378351452082826 [ 6963.002551] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6963.013105] LustreError: 236903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6963.349690] Lustre: server umount lustre-MDT0001 complete [ 6972.033684] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6974.604349] Key type lgssc unregistered [ 6974.926882] LNet: 238196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6974.935205] LNetError: 238196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6974.949819] LNet: Removed LNI 192.168.203.122@tcp [ 6975.670220] Key type .llcrypt unregistered [ 6975.671791] Key type ._llcrypt unregistered [ 6995.828752] Key type ._llcrypt registered [ 6995.833651] Key type .llcrypt registered [ 6995.977742] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6996.996631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6997.107438] alg: No test for adler32 (adler32-zlib) [ 6998.152245] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 6998.401856] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7000.088183] Key type lgssc registered [ 7001.226869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7010.512812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7010.536319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.194310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7017.079384] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7021.545906] LustreError: 239641:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7021.572282] LustreError: 239641:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7026.047747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.261611] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7026.299715] Lustre: 239631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7026.325639] Lustre: Permanently deactivating lustre-MDT0001 [ 7026.370363] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7026.728100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7027.859896] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7027.864948] LustreError: 239672:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7027.884807] LustreError: 239672:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7027.901708] LustreError: 239672:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7031.344664] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7039.380932] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7046.211715] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7053.238733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.625069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7056.690455] LustreError: 241483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7056.719341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7061.120746] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7061.993689] LustreError: 241639:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.107835] LustreError: 241485:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7069.238905] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7072.226045] LustreError: 241483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.250212] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7075.612161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7075.803648] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7081.046219] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7081.050305] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7083.661398] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7090.867876] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7093.959265] Lustre: Permanently reactivating lustre-MDT0001 [ 7093.978776] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7093.986286] Lustre: Skipped 2 previous similar messages [ 7096.545381] Lustre: lustre-MDT0001: Client 12741f93-2cf8-4e7c-8300-be58ce80649f (at 192.168.203.22@tcp) reconnecting [ 7103.288375] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7104.960767] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7111.333822] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7111.516867] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7116.066209] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7116.356581] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7118.995099] Lustre: Permanently deactivating lustre-MDT0001 [ 7119.013974] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7119.023460] Lustre: Skipped 2 previous similar messages [ 7128.690708] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7137.250552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7137.259507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7137.267526] Lustre: Skipped 3 previous similar messages [ 7137.278953] Lustre: Skipped 3 previous similar messages [ 7142.382837] LustreError: 239642:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.421260] LustreError: 239642:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 7142.535812] Lustre: server umount lustre-MDT0000 complete [ 7146.072991] LustreError: 239622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843413 with bad export cookie 8074275121111333684 [ 7146.079679] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7146.088256] LustreError: 239622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7146.616784] Lustre: server umount lustre-MDT0001 complete [ 7152.675054] Lustre: server umount lustre-OST0000 complete [ 7164.502743] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:10:29 (1787843429) [ 7176.571168] Lustre: server umount lustre-OST0001 complete [ 7184.211925] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7196.966438] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7234.492925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7234.744760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7234.754036] Lustre: Skipped 2 previous similar messages [ 7234.777225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7234.887227] Lustre: lustre-MDT0000: new disk, initializing [ 7235.014432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7235.030373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7239.976390] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7250.227029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7250.374791] Lustre: 247952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7250.437771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7250.448396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7250.449197] Lustre: Skipped 1 previous similar message [ 7250.534434] Lustre: lustre-MDT0001: new disk, initializing [ 7250.636754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7250.682273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7250.701260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7254.853637] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7264.915865] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7270.308470] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7276.601616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7276.914526] Lustre: lustre-OST0000: new disk, initializing [ 7276.920984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7276.927205] Lustre: 249921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7277.004875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7283.213442] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7283.798759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7283.813209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7283.907843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7293.053813] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7295.846767] LustreError: 247960:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7305.651300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.858564] Lustre: lustre-OST0001: new disk, initializing [ 7305.874711] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7305.891793] Lustre: 251034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7305.912119] LustreError: 247960:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7305.954044] LustreError: 247959:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7314.558116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7314.566096] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7314.717223] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7314.781700] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7315.955614] LustreError: 247959:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7325.055868] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7326.459786] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7330.297053] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7330.329741] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7330.333051] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7330.346648] Lustre: Skipped 1 previous similar message [ 7334.447484] Lustre: server umount lustre-OST0001 complete [ 7335.422076] LustreError: 251295:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.452119] LustreError: 251295:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7355.883737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7355.885638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.892808] LustreError: Skipped 1 previous similar message [ 7355.894495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7355.901136] Lustre: Skipped 2 previous similar messages [ 7357.715823] Lustre: server umount lustre-OST0000 complete [ 7363.555553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7363.568753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7363.581520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.589466] Lustre: Skipped 1 previous similar message [ 7367.974690] Lustre: server umount lustre-MDT0000 complete [ 7371.234629] LustreError: 250689:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7371.254587] LustreError: 250689:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 7371.520867] LustreError: 248746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843639 with bad export cookie 8074275121111336722 [ 7371.530602] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7371.542552] LustreError: 248746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7378.103876] Lustre: server umount lustre-MDT0001 complete [ 7389.446389] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7392.611431] Key type lgssc unregistered [ 7392.971588] LNet: 253073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7392.982455] LNetError: 253073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7393.014562] LNet: Removed LNI 192.168.203.122@tcp [ 7394.436497] Key type .llcrypt unregistered [ 7394.440110] Key type ._llcrypt unregistered [ 7440.693741] Key type ._llcrypt registered [ 7440.702294] Key type .llcrypt registered [ 7440.905877] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7442.663749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7442.758094] alg: No test for adler32 (adler32-zlib) [ 7443.970383] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 7444.257893] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7445.888205] Key type lgssc registered [ 7447.259962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7455.461428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7455.477564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.706290] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7456.731498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7456.900989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.842376] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7469.091514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.206236] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7469.236781] Lustre: 255282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7469.509349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.626964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7480.526821] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7485.806309] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7492.509064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7492.711652] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7492.911349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7498.813654] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7498.889386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7506.893657] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7560.162078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7560.177126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.200538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7560.679388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7560.690645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.708585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7565.365497] Lustre: server umount lustre-OST0000 complete [ 7569.862484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7582.995223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7583.483701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7591.207855] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7599.470918] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7600.888885] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7600.943237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7600.949715] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7642.086602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7642.089472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7642.122096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7642.133205] Lustre: Skipped 1 previous similar message [ 7646.887970] Lustre: server umount lustre-OST0000 complete [ 7652.321901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7652.325688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7652.337111] Lustre: Skipped 2 previous similar messages [ 7652.349307] Lustre: Skipped 1 previous similar message [ 7656.918508] Lustre: server umount lustre-MDT0000 complete [ 7661.018168] LustreError: 255274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843928 with bad export cookie 9378737450415400597 [ 7661.021150] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7661.041253] LustreError: 255274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7661.542661] Lustre: server umount lustre-MDT0001 complete [ 7672.083727] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7674.631515] Key type lgssc unregistered [ 7674.918540] LNet: 260708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7674.922612] LNetError: 260708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7674.938545] LNet: Removed LNI 192.168.203.122@tcp [ 7675.737475] Key type .llcrypt unregistered [ 7675.741252] Key type ._llcrypt unregistered [ 7698.750865] Key type ._llcrypt registered [ 7698.753073] Key type .llcrypt registered [ 7698.852652] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7699.738469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7699.774335] alg: No test for adler32 (adler32-zlib) [ 7700.825471] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 7701.083198] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7702.768533] Key type lgssc registered [ 7703.817330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7711.201050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7711.224498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7712.750238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.833374] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7722.979880] LustreError: 262121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7723.001268] LustreError: 262121:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7724.404171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.753694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7728.340815] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7734.147171] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7739.210622] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7744.714870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.004022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7750.271545] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7757.206158] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7758.323163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7780.844961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7780.867342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7780.888927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7783.909152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7783.912834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7783.928626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7783.961471] Lustre: Skipped 1 previous similar message [ 7786.075896] Lustre: server umount lustre-OST0000 complete [ 7791.587616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7791.592940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.614129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7794.145926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7794.149188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.743974] Lustre: server umount lustre-MDT0000 complete [ 7799.271466] LustreError: 262122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.272056] LustreError: 262855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844066 with bad export cookie 6281584399581472820 [ 7799.273978] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7799.321988] LustreError: 262855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7799.612882] Lustre: server umount lustre-MDT0001 complete [ 7809.252226] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7811.639555] Key type lgssc unregistered [ 7811.937244] LNet: 266428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7811.949397] LNetError: 266428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7811.973463] LNet: Removed LNI 192.168.203.122@tcp [ 7812.746426] Key type .llcrypt unregistered [ 7812.749110] Key type ._llcrypt unregistered [ 7827.316959] Key type ._llcrypt registered [ 7827.318752] Key type .llcrypt registered [ 7827.441464] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7828.424405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7828.445892] alg: No test for adler32 (adler32-zlib) [ 7829.493438] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 7829.723892] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7831.408253] Key type lgssc registered [ 7832.496358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7840.018533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7840.044580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7841.507390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7845.248963] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7852.011934] LustreError: 267628:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7852.036130] LustreError: 267628:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7853.835357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.229994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7858.694186] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7865.536482] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7871.347873] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7878.592956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7879.060721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7886.432992] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7887.301191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7895.085493] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7910.885732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7910.899962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7910.925109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7912.938241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7912.948289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7912.986501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7914.570503] Lustre: server umount lustre-OST0000 complete [ 7920.613355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7920.629274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7920.665676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7920.674981] Lustre: Skipped 1 previous similar message [ 7923.174777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7923.176901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7923.214125] Lustre: Skipped 1 previous similar message [ 7925.319829] Lustre: server umount lustre-MDT0000 complete [ 7928.306178] LustreError: 267624:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.320396] LustreError: 267624:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7929.513466] LustreError: 267609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844197 with bad export cookie 7589429181778943029 [ 7929.536395] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7929.537903] LustreError: 267609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7930.153620] Lustre: server umount lustre-MDT0001 complete [ 7942.378360] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7945.343670] Key type lgssc unregistered [ 7945.596227] LNet: 271310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7945.611355] LNetError: 271310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7945.633846] LNet: Removed LNI 192.168.203.122@tcp [ 7946.679662] Key type .llcrypt unregistered [ 7946.682775] Key type ._llcrypt unregistered [ 7972.476754] Key type ._llcrypt registered [ 7972.479268] Key type .llcrypt registered [ 7972.570872] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7973.707835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7973.761763] alg: No test for adler32 (adler32-zlib) [ 7974.918162] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 7975.217471] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7976.856289] Key type lgssc registered [ 7978.002891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7985.824774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7985.851211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.256867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.690280] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7996.392078] LustreError: 272728:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.436510] LustreError: 272728:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7999.599918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.934336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.275971] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8011.529862] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8016.856281] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8023.352244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8023.784130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.522129] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8032.004044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8037.747642] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8060.900072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8060.903800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8060.927796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8062.945916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8062.950876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8062.951242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8062.962266] Lustre: Skipped 1 previous similar message [ 8066.317344] Lustre: server umount lustre-OST0000 complete [ 8071.650077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8071.664808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8071.687404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8076.254571] Lustre: server umount lustre-MDT0000 complete [ 8078.310416] LustreError: 272725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8078.325258] LustreError: 272725:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8079.605700] LustreError: 272710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844347 with bad export cookie 8625679569519155270 [ 8079.607584] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8079.620480] LustreError: 272710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8079.877783] Lustre: server umount lustre-MDT0001 complete [ 8090.215340] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8092.523783] Key type lgssc unregistered [ 8092.770687] LNet: 277038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8092.781502] LNetError: 277038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8092.803308] LNet: Removed LNI 192.168.203.122@tcp [ 8093.499369] Key type .llcrypt unregistered [ 8093.503456] Key type ._llcrypt unregistered [ 8109.493680] Key type ._llcrypt registered [ 8109.500378] Key type .llcrypt registered [ 8109.608649] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8110.653749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8110.696262] alg: No test for adler32 (adler32-zlib) [ 8111.768389] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 8111.999180] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8113.648165] Key type lgssc registered [ 8115.151552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8122.605713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8122.626347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.203409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8128.772543] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8133.601912] LustreError: 278272:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8133.616390] LustreError: 278272:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8138.726514] LustreError: 278274:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8138.967886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.452041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8144.583963] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8152.920334] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8159.769332] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8167.456828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8168.035985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.015318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8174.945578] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8182.079164] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8199.659055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8199.674255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8199.675770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8199.701061] Lustre: Skipped 1 previous similar message [ 8202.374535] Lustre: server umount lustre-OST0000 complete [ 8209.902154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.916196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8209.926086] Lustre: Skipped 2 previous similar messages [ 8212.844430] Lustre: server umount lustre-MDT0000 complete [ 8215.014552] LustreError: 278276:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8215.039446] LustreError: 278276:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8216.878034] LustreError: 278238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844484 with bad export cookie 18185021492706065678 [ 8216.885323] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8216.896798] LustreError: 278238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8217.327739] Lustre: server umount lustre-MDT0001 complete [ 8229.616416] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8232.785420] Key type lgssc unregistered [ 8233.128225] LNet: 281976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8233.138952] LNetError: 281976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8234.214767] LNet: Removed LNI 192.168.203.122@tcp [ 8234.719697] Key type .llcrypt unregistered [ 8234.721862] Key type ._llcrypt unregistered [ 8247.079913] Key type ._llcrypt registered [ 8247.083267] Key type .llcrypt registered [ 8247.222238] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8268.444190] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8286.471289] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8287.541634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8287.567252] alg: No test for adler32 (adler32-zlib) [ 8288.550443] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 8288.825151] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8290.520278] Key type lgssc registered [ 8291.640320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8331.614127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8331.650891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8333.003631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8333.059434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8333.196138] Lustre: lustre-MDT0000: new disk, initializing [ 8333.351052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.409688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8339.490494] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8351.166169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8351.374548] Lustre: 285532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8351.490850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8351.494984] Lustre: Skipped 1 previous similar message [ 8351.551727] Lustre: lustre-MDT0001: new disk, initializing [ 8351.692463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8351.757692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8351.775915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8356.634775] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8366.030531] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8371.574328] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8377.619305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.917281] Lustre: lustre-OST0000: new disk, initializing [ 8377.921103] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8377.925957] Lustre: 287501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8378.056970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8383.527214] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8388.169132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8388.179019] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8388.306167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8392.314497] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8397.219288] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8397.445494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8402.079677] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8402.274558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8403.951028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.965756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8403.967883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8409.067455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8409.068221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8409.101385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8414.185029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8414.190600] Lustre: Skipped 1 previous similar message [ 8418.274607] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8418.532925] Lustre: server umount lustre-OST0000 complete [ 8423.396177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8423.412204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.451877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8423.456633] Lustre: Skipped 1 previous similar message [ 8429.537794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8429.547390] Lustre: Skipped 3 previous similar messages [ 8437.216410] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8437.575909] Lustre: server umount lustre-MDT0000 complete [ 8441.949528] LustreError: 285524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844709 with bad export cookie 3400543424935624709 [ 8441.957531] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8441.964716] LustreError: 285524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8442.352068] Lustre: server umount lustre-MDT0001 complete [ 8450.582937] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:31:55 (1787844715) [ 8457.823912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8458.357320] LustreError: 289915:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8458.372979] LustreError: 289915:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8458.478088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8463.329315] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8463.849789] LustreError: 289916:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8468.961753] LustreError: 289915:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8473.281360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8473.711116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8478.611265] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8485.779094] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8491.219299] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8498.607657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8499.064625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8506.070716] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8513.575474] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8528.873401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8528.878161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8528.884798] Lustre: Skipped 2 previous similar messages [ 8528.906608] Lustre: Skipped 3 previous similar messages [ 8532.284890] Lustre: server umount lustre-OST0000 complete [ 8539.108055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.126618] Lustre: Skipped 1 previous similar message [ 8542.523556] Lustre: server umount lustre-MDT0000 complete [ 8544.231848] LustreError: 289915:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8544.267139] LustreError: 289915:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8546.263855] LustreError: 289895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844813 with bad export cookie 3400543424935625591 [ 8546.271836] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8546.284714] LustreError: 289895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8546.964229] Lustre: server umount lustre-MDT0001 complete [ 8559.459537] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8563.389192] Key type lgssc unregistered [ 8563.828587] LNet: 293466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8563.836739] LNetError: 293466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8563.856207] LNet: Removed LNI 192.168.203.122@tcp [ 8564.741788] Key type .llcrypt unregistered [ 8564.744193] Key type ._llcrypt unregistered [ 8590.523405] Key type ._llcrypt registered [ 8590.526558] Key type .llcrypt registered [ 8590.622941] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8607.066451] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8608.050482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8608.147492] alg: No test for adler32 (adler32-zlib) [ 8609.236480] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 8609.485969] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8611.200168] Key type lgssc registered [ 8612.112379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8655.085153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8655.106977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.526806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8656.600307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8656.810634] Lustre: lustre-MDT0000: new disk, initializing [ 8656.911849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8656.937437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8661.739563] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8674.361530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.557622] Lustre: 297291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8674.652917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8674.657442] Lustre: Skipped 1 previous similar message [ 8674.758394] Lustre: lustre-MDT0001: new disk, initializing [ 8674.850161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8674.897477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8674.905968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8678.855722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8688.085537] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8693.826739] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8700.882304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8701.350066] Lustre: lustre-OST2710: new disk, initializing [ 8701.359235] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8701.371880] Lustre: 299260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8701.503859] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8707.165429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8707.185083] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8707.319475] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8707.747461] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8717.923993] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8724.700896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8724.887952] Lustre: lustre-OST03e8: new disk, initializing [ 8724.892609] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8724.903142] Lustre: 300324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8725.021861] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8730.754710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8730.768807] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8730.872969] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8731.389484] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8739.618247] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8748.091292] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8748.349814] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8753.579573] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8753.836744] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8759.651377] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8759.950886] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8764.979277] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8765.128663] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8772.067683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8772.071119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8772.074520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8772.090535] Lustre: Skipped 1 previous similar message [ 8772.378979] Lustre: server umount lustre-MDT0000 complete [ 8776.394900] LustreError: 297283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845044 with bad export cookie 15757066956359187744 [ 8776.408918] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8776.428659] LustreError: 297283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8777.009989] Lustre: server umount lustre-MDT0001 complete [ 8791.042410] Lustre: server umount lustre-OST2710 complete [ 8792.995249] Lustre: 295091:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845044/real 1787845044] req@000000003098d042 x1874691229814272/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787845060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8793.016380] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.025346] Lustre: Skipped 2 previous similar messages [ 8794.392486] Lustre: server umount lustre-OST03e8 complete [ 8813.054904] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8823.378906] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8857.977836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8858.246365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8858.273240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8858.350322] Lustre: lustre-MDT0000: new disk, initializing [ 8858.457849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8858.470850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8862.860426] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8873.412586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8873.622109] Lustre: 305571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8873.745827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8873.754130] Lustre: Skipped 1 previous similar message [ 8873.884468] Lustre: lustre-MDT0001: new disk, initializing [ 8874.003766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8874.049870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8874.064164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8879.317684] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8889.477051] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8894.618944] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8902.002767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.333437] Lustre: lustre-OST0000: new disk, initializing [ 8902.338675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8902.344168] Lustre: 307541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8902.429291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8908.128809] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8909.878467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8909.888668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8909.974315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8916.361522] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8920.211677] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8920.394610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8924.770906] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8925.050417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8930.805185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8930.806283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8930.811074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8930.811082] Lustre: Skipped 3 previous similar messages [ 8930.820824] Lustre: Skipped 1 previous similar message [ 8930.852732] LustreError: Skipped 1 previous similar message [ 8933.368877] Lustre: server umount lustre-OST0000 complete [ 8940.514751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8940.544166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8940.629657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8940.656070] Lustre: Skipped 1 previous similar message [ 8945.365670] Lustre: server umount lustre-MDT0000 complete [ 8952.517615] LustreError: 307540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845220 with bad export cookie 15757066956359189424 [ 8952.519741] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8952.544507] LustreError: 307540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8952.797910] Lustre: server umount lustre-MDT0001 complete [ 8963.618426] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:40:29 (1787845229) [ 8965.317590] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8967.044322] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:40:32 (1787845232) [ 8970.760342] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8996.980809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.281583] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8997.322174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8997.470591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.391489] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9008.492944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9008.603475] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9008.635516] Lustre: 311398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9013.484953] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9034.943103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9035.199798] Lustre: Denying initial registration attempt from nid 192.168.203.122@tcp, specified as failover [ 9035.215739] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9035.241279] LustreError: 312854:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9035.254875] LustreError: 312854:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9035.268472] LustreError: 312854:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9035.377957] Lustre: server umount lustre-OST0000 complete [ 9035.381709] LustreError: 312854:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9041.890267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9041.907090] Lustre: Skipped 1 previous similar message [ 9041.915827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9041.930939] Lustre: Skipped 3 previous similar messages [ 9047.013136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9047.021133] Lustre: Skipped 1 previous similar message [ 9056.224147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9056.406465] Lustre: server umount lustre-MDT0000 complete [ 9057.253409] LustreError: 311406:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9057.285904] LustreError: 311406:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 9060.524467] LustreError: 311392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845328 with bad export cookie 15757066956359190306 [ 9060.542941] LustreError: 311392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9060.547639] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9061.165826] Lustre: server umount lustre-MDT0001 complete [ 9073.250358] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9077.628785] Key type lgssc unregistered [ 9077.891628] LNet: 313978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9077.900679] LNetError: 313978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9077.919193] LNet: Removed LNI 192.168.203.122@tcp [ 9078.804219] Key type .llcrypt unregistered [ 9078.819808] Key type ._llcrypt unregistered [ 9090.124716] Key type ._llcrypt registered [ 9090.129281] Key type .llcrypt registered [ 9090.227759] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9091.534111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9091.588536] alg: No test for adler32 (adler32-zlib) [ 9092.722939] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 9092.944957] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9094.656173] Key type lgssc registered [ 9095.786701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9137.696517] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9145.897272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9145.915456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9147.298909] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9147.340683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9147.514580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9152.187121] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9161.926864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9162.118570] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9162.180577] Lustre: 316722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9162.532703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9168.610726] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9175.847698] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9180.924128] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9200.669289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9200.960595] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9201.307495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9207.632232] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9216.082357] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9222.121476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9222.132449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.160173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9222.632986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9222.648238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.674820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9224.820958] Lustre: server umount lustre-OST0000 complete [ 9229.283142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9229.293619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9229.311597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9232.872629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9232.876109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9234.692522] Lustre: server umount lustre-MDT0000 complete [ 9238.897414] LustreError: 316713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845506 with bad export cookie 11739679826599943774 [ 9238.905101] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9239.330239] Lustre: server umount lustre-MDT0001 complete [ 9252.870366] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9255.828580] Key type lgssc unregistered [ 9256.046258] LNet: 320300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9256.054268] LNetError: 320300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9256.075434] LNet: Removed LNI 192.168.203.122@tcp [ 9256.898926] Key type .llcrypt unregistered [ 9256.902430] Key type ._llcrypt unregistered [ 9280.590946] Key type ._llcrypt registered [ 9280.593386] Key type .llcrypt registered [ 9280.703861] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9282.024289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9282.062920] alg: No test for adler32 (adler32-zlib) [ 9283.293944] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 9283.570962] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9285.272214] Key type lgssc registered [ 9286.199090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9294.312711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9294.364836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9295.975456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9301.139439] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9305.578665] LustreError: 321719:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9305.616273] LustreError: 321719:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9309.251103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.621516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9314.906632] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9322.685303] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9328.880367] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9337.545621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9338.076072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9344.658912] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9353.623804] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9364.965859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.968468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9364.993672] Lustre: Skipped 1 previous similar message [ 9365.013076] Lustre: Skipped 2 previous similar messages [ 9366.884332] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 9369.435511] Lustre: server umount lustre-MDT0000 complete [ 9370.090989] LustreError: 322447:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.130596] LustreError: 322447:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9371.974579] LustreError: 323113:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9374.063070] LustreError: 321699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845641 with bad export cookie 4997624700159932551 [ 9374.072572] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9374.081549] LustreError: 321699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9375.202032] LustreError: 321714:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9375.202343] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.278329] Lustre: Skipped 1 previous similar message [ 9375.292146] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9379.234175] LustreError: 321715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9379.237160] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9379.260645] LustreError: 321715:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9379.290749] Lustre: Skipped 1 previous similar message [ 9380.870319] Lustre: server umount lustre-MDT0001 complete [ 9387.608906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9397.680862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9398.756429] LustreError: 321030:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001616d641 x1874691936528000/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9399.198299] LustreError: 323554:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9399.312984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9399.400963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9404.139511] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9409.514504] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9412.582189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9413.043620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9417.817409] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9418.220751] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9425.316963] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9427.134730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9433.171565] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9434.707456] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9440.226856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9440.240761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9440.254940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9443.820190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9443.823851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9445.655138] Lustre: server umount lustre-OST0000 complete [ 9454.052520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9454.053549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9454.059795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9454.082068] Lustre: Skipped 1 previous similar message [ 9454.097456] Lustre: Skipped 3 previous similar messages [ 9464.289578] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9464.560336] Lustre: server umount lustre-MDT0000 complete [ 9468.843887] LustreError: 324504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845736 with bad export cookie 4997624700159940244 [ 9468.848131] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9468.852676] LustreError: 324504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9469.344672] Lustre: server umount lustre-MDT0001 complete [ 9482.106512] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9485.999669] Key type lgssc unregistered [ 9486.471539] LNet: 327989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9486.507253] LNetError: 327989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9486.540336] LNet: Removed LNI 192.168.203.122@tcp [ 9487.738991] Key type .llcrypt unregistered [ 9487.741759] Key type ._llcrypt unregistered [ 9514.724150] Key type ._llcrypt registered [ 9514.731369] Key type .llcrypt registered [ 9514.848584] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9515.951994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9516.102222] alg: No test for adler32 (adler32-zlib) [ 9517.184812] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 9517.394357] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9519.040206] Key type lgssc registered [ 9520.287441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9527.948481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9527.971327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9529.372355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9533.839822] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9539.561165] LustreError: 329407:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9539.584574] LustreError: 329407:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9542.326576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9542.743874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9547.428209] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9554.334028] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9559.750741] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9563.105824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9563.115363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.135685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9565.155462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9565.164096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9568.290376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9568.297557] Lustre: Skipped 2 previous similar messages [ 9568.934279] Lustre: server umount lustre-MDT0000 complete [ 9572.374646] LustreError: 329388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845840 with bad export cookie 10097881942357837250 [ 9572.382716] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9572.397644] LustreError: 329388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9572.693816] Lustre: server umount lustre-MDT0001 complete [ 9587.268301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9587.665984] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9587.694983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9587.901034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9592.142633] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9602.411593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9602.509089] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9602.600711] Lustre: 331994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9603.051253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9610.176772] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9620.279607] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9627.436614] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9638.184221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9638.441330] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9638.450639] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9638.463777] LustreError: 333880:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9638.469898] LustreError: 333880:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9638.485727] LustreError: 333880:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9638.571424] Lustre: server umount lustre-OST0000 complete [ 9638.575098] LustreError: 333880:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9647.913773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9648.159538] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9648.382319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9654.018425] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9655.439988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9660.678583] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9668.281852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9668.371260] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9668.461995] Lustre: lustre-OST0001: new disk, initializing [ 9668.465764] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9668.470993] Lustre: 335426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9668.560711] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9673.821182] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9674.303528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9674.318059] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9674.405559] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9681.303873] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9684.975937] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9684.977030] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9684.983336] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9684.996017] LustreError: Skipped 1 previous similar message [ 9685.042036] Lustre: Skipped 1 previous similar message [ 9690.087239] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9690.095879] Lustre: Skipped 2 previous similar messages [ 9698.272226] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9698.480790] Lustre: server umount lustre-OST0001 complete [ 9700.332686] LustreError: 335811:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9700.370273] LustreError: 335811:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9705.445543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9705.456758] LustreError: 335811:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9705.459045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9705.459052] Lustre: Skipped 2 previous similar messages [ 9705.466984] Lustre: Skipped 1 previous similar message [ 9705.492437] LustreError: 335811:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9708.829321] Lustre: server umount lustre-OST0000 complete [ 9715.681339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9715.685083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9715.714094] Lustre: Skipped 1 previous similar message [ 9719.450026] Lustre: server umount lustre-MDT0000 complete [ 9720.802409] LustreError: 332007:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9720.824753] LustreError: 332007:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9723.004870] LustreError: 331986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845990 with bad export cookie 10097881942357837796 [ 9723.011513] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9723.020130] LustreError: 331986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9723.403712] Lustre: server umount lustre-MDT0001 complete [ 9734.905782] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9737.890154] Key type lgssc unregistered [ 9738.170954] LNet: 337388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9738.182481] LNetError: 337388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9739.260930] LNet: Removed LNI 192.168.203.122@tcp [ 9740.293124] Key type .llcrypt unregistered [ 9740.296664] Key type ._llcrypt unregistered [ 9809.562328] Key type ._llcrypt registered [ 9809.566035] Key type .llcrypt registered [ 9809.666575] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 9827.099495] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9828.900814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9828.926737] alg: No test for adler32 (adler32-zlib) [ 9830.134172] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 9830.518526] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9832.331256] Key type lgssc registered [ 9833.496236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9870.943176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9870.966617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9872.262160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9872.310782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9872.427376] Lustre: lustre-MDT0000: new disk, initializing [ 9872.516988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9872.536883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9877.073513] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9887.360907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9887.526709] Lustre: 342323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9887.583471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9887.588641] Lustre: Skipped 1 previous similar message [ 9887.670794] Lustre: lustre-MDT0001: new disk, initializing [ 9887.691406] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9887.785572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9887.822898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9887.835416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9892.756435] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9901.728612] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9907.056748] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9912.810370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9913.095372] Lustre: lustre-OST0000: new disk, initializing [ 9913.102590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9913.118532] Lustre: 344293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9913.208408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.344585] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9920.561206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9920.576128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9920.680882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9927.772843] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9933.387341] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9933.566783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9937.840846] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9938.051629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9941.476455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.481374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9941.485770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9941.496280] Lustre: Skipped 1 previous similar message [ 9946.724188] Lustre: server umount lustre-OST0000 complete [ 9951.718894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9951.723146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9951.744442] Lustre: Skipped 1 previous similar message [ 9951.763148] Lustre: Skipped 2 previous similar messages [ 9956.845585] LustreError: 342336:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.870719] LustreError: 342336:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9956.942425] Lustre: server umount lustre-MDT0000 complete [ 9960.462984] LustreError: 342316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846228 with bad export cookie 8132278727222011018 [ 9960.466832] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9960.474340] LustreError: 342316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9960.918862] Lustre: server umount lustre-MDT0001 complete [ 9968.864543] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:57:14 (1787846234) [ 9976.225854] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:57:21 (1787846241) [ 9984.662051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9985.164032] LustreError: 347051:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.181938] LustreError: 347051:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9985.230126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9989.432771] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9990.635091] LustreError: 347052:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9994.803204] LustreError: 347051:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.170706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9998.596654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10003.630252] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10010.716734] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10016.350425] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10023.044114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10023.335165] LustreError: 348871:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10023.528937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10032.203379] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10041.456911] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10049.814431] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10051.684818] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10053.255543] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10054.837911] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10056.684070] Lustre: Failing over lustre-MDT0000 [10056.992346] Lustre: server umount lustre-MDT0000 complete [10059.639834] LustreError: 349593:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.258516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10060.264715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10060.291366] Lustre: Skipped 1 previous similar message [10060.310202] LustreError: Skipped 1 previous similar message [10075.449680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.621432] Lustre: 340548:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846327/real 1787846327] req@0000000052f19313 x1874692510127616/t0(0) o400->MGC192.168.203.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787846343 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10075.642834] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10075.905537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10075.981217] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10076.035742] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10080.245737] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10081.263133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10081.305408] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10081.373635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10089.963862] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10091.720891] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10095.634660] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10097.361131] Lustre: Failing over lustre-MDT0000 [10097.714648] Lustre: server umount lustre-MDT0000 complete [10101.593707] LustreError: 347046:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10101.617404] LustreError: 347046:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [10101.729203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10101.738747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10101.759273] Lustre: Skipped 3 previous similar messages [10117.677085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10117.883680] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10118.227247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10118.268895] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10123.075190] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10123.190396] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10123.245607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10123.255927] Lustre: Skipped 2 previous similar messages [10123.280651] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10123.348944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10132.784965] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10134.371308] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10143.717674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10143.727697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10143.740644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10143.772358] Lustre: Skipped 2 previous similar messages [10145.611073] Lustre: server umount lustre-MDT0000 complete [10148.833627] LustreError: 347052:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10148.857459] LustreError: 347052:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 18 previous similar messages [10149.378770] LustreError: 347031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846417 with bad export cookie 8132278727222013727 [10149.392783] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10149.752144] Lustre: server umount lustre-MDT0001 complete [10163.862375] Lustre: server umount lustre-OST0000 complete [10179.672945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10184.660478] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10192.993443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.400517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10193.417990] Lustre: Skipped 1 previous similar message [10197.674362] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10205.751424] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10211.261708] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10218.001693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10223.986332] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10224.571144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10231.905785] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10238.734726] Lustre: server umount lustre-OST0000 complete [10244.577275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10244.589912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10244.625790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10244.636367] Lustre: Skipped 2 previous similar messages [10255.328744] Lustre: 340545:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846507/real 1787846507] req@0000000062793fc1 x1874692510195456/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787846523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10255.335296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10255.365768] Lustre: 340545:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10255.374312] Lustre: Skipped 5 previous similar messages [10256.352874] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10256.570921] Lustre: server umount lustre-MDT0000 complete [10260.075845] LustreError: 353371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846527 with bad export cookie 8132278727222014476 [10260.088826] LustreError: 353371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10260.089078] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10260.497823] Lustre: server umount lustre-MDT0001 complete [10272.803812] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [10275.726901] Key type lgssc unregistered [10276.065089] LNet: 356937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10276.081288] LNetError: 356937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10276.097598] LNet: Removed LNI 192.168.203.122@tcp [10276.977469] Key type .llcrypt unregistered [10276.984208] Key type ._llcrypt unregistered [10305.086873] Key type ._llcrypt registered [10305.091166] Key type .llcrypt registered [10305.198434] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [10306.490267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10306.561877] alg: No test for adler32 (adler32-zlib) [10307.871657] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [10308.280328] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [10310.090493] Key type lgssc registered [10312.187805] Lustre: Echo OBD driver; http://www.lustre.org/ [10323.206555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10323.269657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.014207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10331.584555] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10335.204240] LustreError: 358455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.226487] LustreError: 358455:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10340.331589] LustreError: 358456:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.895439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.545628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10346.966202] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10354.145254] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10360.993997] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10368.154765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10368.626296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10369.701595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10375.419684] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10384.138833] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10402.787214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10402.791709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10402.793336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10402.799786] Lustre: Skipped 2 previous similar messages [10404.925934] Lustre: server umount lustre-MDT0000 complete [10407.909720] LustreError: 358451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.937282] LustreError: 358451:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10408.824650] LustreError: 358437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846676 with bad export cookie 13872675779280144916 [10408.830491] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10408.838664] LustreError: 358437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10409.155247] Lustre: server umount lustre-MDT0001 complete [10424.117763] Lustre: server umount lustre-OST0000 complete [10439.525499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10440.130702] LustreError: 362188:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.197788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10444.995455] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10445.282886] LustreError: 362189:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10454.497657] LustreError: 362188:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10454.526144] LustreError: 362188:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10454.729819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10455.199929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10460.098809] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10468.412082] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10473.507137] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10480.579791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10480.897970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10485.005727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10488.177734] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10497.007526] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10509.799386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10509.807748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10509.832822] Lustre: Skipped 1 previous similar message [10509.835914] Lustre: Skipped 2 previous similar messages [10514.923412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10514.936972] Lustre: Skipped 4 previous similar messages [10515.242716] Lustre: server umount lustre-MDT0000 complete [10518.506802] LustreError: 362168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846786 with bad export cookie 13872675779280146029 [10518.518799] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10518.526592] LustreError: 362168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10518.784709] Lustre: server umount lustre-MDT0001 complete [10533.109792] Lustre: server umount lustre-OST0000 complete [10549.614617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10550.086719] LustreError: 366102:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10550.179079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.064670] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10564.089667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10569.617962] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10578.184358] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10584.435075] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10592.079451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10592.517093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10592.532708] Lustre: Skipped 1 previous similar message [10594.567442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10600.336092] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10608.345797] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10615.267842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.274238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10615.292201] Lustre: Skipped 1 previous similar message [10615.315816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10616.408183] Lustre: server umount lustre-OST0000 complete [10625.506596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10625.512747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10625.523249] Lustre: Skipped 2 previous similar messages [10625.529284] Lustre: Skipped 2 previous similar messages [10635.751660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10635.770337] Lustre: Skipped 3 previous similar messages [10636.256127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10636.765578] Lustre: server umount lustre-MDT0000 complete [10640.872499] LustreError: 366097:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10640.907259] LustreError: 366097:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10642.677080] LustreError: 366083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846910 with bad export cookie 13872675779280147051 [10642.685570] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.687481] LustreError: 366083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10643.255800] Lustre: server umount lustre-MDT0001 complete [10656.966811] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [10659.818328] Key type lgssc unregistered [10660.050566] LNet: 369639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10660.058632] LNetError: 369639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10660.076217] LNet: Removed LNI 192.168.203.122@tcp [10660.728540] Key type .llcrypt unregistered [10660.731716] Key type ._llcrypt unregistered [10687.694405] Key type ._llcrypt registered [10687.696877] Key type .llcrypt registered [10687.854944] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [10689.520456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10689.557944] alg: No test for adler32 (adler32-zlib) [10690.640175] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [10690.871240] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [10692.520159] Key type lgssc registered [10693.800466] Lustre: Echo OBD driver; http://www.lustre.org/ [10700.784522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10700.797710] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10700.808256] LustreError: 371080:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10700.841854] LustreError: 371080:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10700.854648] LustreError: 371080:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10700.866309] LustreError: 371080:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10700.876247] LustreError: 371080:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10708.524515] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10708.530630] LustreError: 371477:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10708.549917] LustreError: 371477:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10708.556324] LustreError: 371477:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10708.562538] LustreError: 371477:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10708.570679] LustreError: 371477:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10727.598068] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [10730.670286] Key type lgssc unregistered [10730.978695] LNet: 372483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10730.990034] LNetError: 372483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10731.021420] LNet: Removed LNI 192.168.203.122@tcp [10731.763460] Key type .llcrypt unregistered [10731.767986] Key type ._llcrypt unregistered [10752.011638] Key type ._llcrypt registered [10752.014100] Key type .llcrypt registered [10752.139648] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [10773.163957] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [10775.551403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10775.591423] alg: No test for adler32 (adler32-zlib) [10776.780628] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [10777.127297] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [10778.848177] Key type lgssc registered [10779.911992] Lustre: Echo OBD driver; http://www.lustre.org/ [10820.996969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10821.043213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10822.175979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10822.221883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10822.273222] Lustre: lustre-MDT0000: new disk, initializing [10822.317179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10822.327412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10827.874168] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10839.466721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10839.648533] Lustre: 375646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10839.712988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10839.718546] Lustre: Skipped 1 previous similar message [10839.822652] Lustre: lustre-MDT0001: new disk, initializing [10839.934046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10839.999094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10840.007492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10844.974610] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10853.953198] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10859.445457] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10866.573779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10866.812841] Lustre: lustre-OST0000: new disk, initializing [10866.817417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10866.827310] Lustre: 377612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10866.937881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10872.531718] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10873.402520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10873.419362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10873.588991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10881.525827] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10885.811408] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10886.089192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10891.879766] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10892.214692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10894.307746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10894.317507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.330282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10894.351683] Lustre: Skipped 1 previous similar message [10894.366643] Lustre: Skipped 1 previous similar message [10899.432521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10899.441919] Lustre: Skipped 1 previous similar message [10900.516843] Lustre: server umount lustre-OST0000 complete [10906.598250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10906.605689] LustreError: Skipped 1 previous similar message [10906.608863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10906.628855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10909.667283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10909.670605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10909.700680] Lustre: Skipped 1 previous similar message [10911.715358] Lustre: server umount lustre-MDT0000 complete [10916.393156] LustreError: 375637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847184 with bad export cookie 8084573436719334273 [10916.397149] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10916.403962] LustreError: 375637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10916.704715] Lustre: server umount lustre-MDT0001 complete [10927.120941] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10929.239097] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:13:14 (1787847194) [10935.929195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10936.332228] LustreError: 380071:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10936.361215] LustreError: 380071:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10936.413896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10941.409899] LustreError: 380072:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.672072] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10946.533986] LustreError: 380071:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.661259] LustreError: 380072:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.353750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10952.861829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10957.410704] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10965.031733] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10971.868770] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10979.969000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10980.479539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10987.440446] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10995.545486] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11003.435641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11003.703931] Lustre: lustre-OST0001: new disk, initializing [11003.708666] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11003.716879] Lustre: 382909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11003.812969] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11010.156677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11010.173070] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11010.283759] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11011.092226] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11021.215320] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11025.889832] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11025.890592] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.900397] LustreError: Skipped 1 previous similar message [11025.930495] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11025.943974] Lustre: Skipped 1 previous similar message [11031.316498] Lustre: server umount lustre-OST0001 complete [11033.439218] LustreError: 383211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11046.377686] LustreError: 383211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11046.420682] LustreError: 383211:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11051.501476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11051.524656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11051.533695] Lustre: Skipped 2 previous similar messages [11051.572932] Lustre: Skipped 4 previous similar messages [11064.288141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11064.465561] Lustre: server umount lustre-OST0000 complete [11070.433677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11070.443261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11070.461432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11070.464621] Lustre: Skipped 4 previous similar messages [11075.616263] Lustre: server umount lustre-MDT0000 complete [11077.095577] LustreError: 380071:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11077.136821] LustreError: 380071:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [11079.863366] LustreError: 380053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847347 with bad export cookie 8084573436719335155 [11079.881481] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11086.491199] Lustre: server umount lustre-MDT0001 complete [11098.821836] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11102.360976] Key type lgssc unregistered [11102.744817] LNet: 384905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11102.760245] LNetError: 384905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11102.788464] LNet: Removed LNI 192.168.203.122@tcp [11103.922982] Key type .llcrypt unregistered [11103.925954] Key type ._llcrypt unregistered [11166.058026] Key type ._llcrypt registered [11166.059625] Key type .llcrypt registered [11166.246463] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11167.423810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11167.538374] alg: No test for adler32 (adler32-zlib) [11169.479035] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [11170.163835] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [11172.168288] Key type lgssc registered [11174.537031] Lustre: Echo OBD driver; http://www.lustre.org/ [11185.399608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11185.436603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.836888] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11186.899634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11187.055128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11193.173721] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11202.979822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.156185] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11203.192507] Lustre: 387706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11203.501541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11208.514430] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11215.998595] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11221.535365] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11224.035267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11224.047955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.059842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11224.549179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.564891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11229.902558] Lustre: server umount lustre-MDT0000 complete [11233.945595] LustreError: 387699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847501 with bad export cookie 9430059545777682259 [11233.957945] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11233.965507] LustreError: 387699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11234.541550] Lustre: server umount lustre-MDT0001 complete [11247.401748] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:18:31 (1787847511) [11256.129518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11256.893337] LustreError: 390217:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11256.933539] LustreError: 390217:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11256.994792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11262.433106] LustreError: 390218:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.566095] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11267.555722] LustreError: 390217:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.324295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11272.676533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11272.824828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11277.423651] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11285.436515] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11291.674407] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11298.663036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11298.847494] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11299.052039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11305.349331] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11312.724850] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11321.579986] Lustre: Permanently deactivating lustre-OST0000 [11321.614557] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11327.546623] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11330.081500] LustreError: 393157:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11330.089336] LustreError: 393157:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11330.094051] LustreError: 393157:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11333.646553] LustreError: 393206:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11333.653830] LustreError: 393206:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11342.216731] Lustre: server umount lustre-OST0000 complete [11345.713165] LustreError: 393456:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11345.717763] LustreError: 393456:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11348.450913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11348.457323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11348.478821] Lustre: Skipped 1 previous similar message [11353.579678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11353.593457] Lustre: Skipped 2 previous similar messages [11362.272241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11362.695249] Lustre: server umount lustre-MDT0000 complete [11363.821472] LustreError: 390212:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11363.847897] LustreError: 390212:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11367.086259] LustreError: 390198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847634 with bad export cookie 9430059545777682840 [11367.094303] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11367.105560] LustreError: 390198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11376.855683] Lustre: server umount lustre-MDT0001 complete [11385.540184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11390.306403] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11393.234861] LustreError: 394616:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11401.042883] Lustre: 394956:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.122@tcp [11403.694804] Lustre: server umount MGS complete [11415.949121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.801235] LustreError: 395680:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11416.834670] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11416.845110] Lustre: Skipped 1 previous similar message [11416.997602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11421.497251] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11427.304456] LustreError: 395680:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11427.324572] LustreError: 395680:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11431.686734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.973373] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11436.834566] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11444.310597] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11449.622659] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11451.591772] Lustre: Permanently reactivating lustre-OST0000 [11451.611552] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11451.617647] Lustre: Skipped 2 previous similar messages [11480.546958] LustreError: 397479:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11489.851297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11490.296927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11490.305488] Lustre: Skipped 1 previous similar message [11497.699626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11506.250368] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11514.851824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11514.875881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11514.894316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11514.896490] Lustre: Skipped 2 previous similar messages [11515.893706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11519.967702] Lustre: server umount lustre-OST0000 complete [11526.130689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11526.166211] Lustre: Skipped 2 previous similar messages [11530.848588] Lustre: server umount lustre-MDT0000 complete [11531.234719] LustreError: 395680:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.258811] LustreError: 395680:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11536.031831] LustreError: 395661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847803 with bad export cookie 9430059545777683890 [11536.039377] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11536.047443] LustreError: 395661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11536.396174] Lustre: server umount lustre-MDT0001 complete [11550.076738] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11553.948517] Key type lgssc unregistered [11554.289643] LNet: 400094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11554.295146] LNetError: 400094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11554.306713] LNet: Removed LNI 192.168.203.122@tcp [11555.685285] Key type .llcrypt unregistered [11555.691266] Key type ._llcrypt unregistered [11573.268046] Key type ._llcrypt registered [11573.271469] Key type .llcrypt registered [11573.428427] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [11588.438790] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11589.528462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11589.623227] alg: No test for adler32 (adler32-zlib) [11590.675655] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [11590.966346] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [11592.648183] Key type lgssc registered [11593.926237] Lustre: Echo OBD driver; http://www.lustre.org/ [11629.006305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11629.032739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11630.329880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11630.374361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11630.472747] Lustre: lustre-MDT0000: new disk, initializing [11630.534224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11630.560904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11635.090439] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11646.229158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.468790] Lustre: 403251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11646.612987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11646.623761] Lustre: Skipped 1 previous similar message [11646.762421] Lustre: lustre-MDT0001: new disk, initializing [11646.917844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11647.028746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11647.059146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11652.627909] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11660.814773] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11666.401367] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11673.326644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11673.584962] Lustre: lustre-OST0000: new disk, initializing [11673.591104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11673.600901] Lustre: 405218:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11673.698677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11679.920421] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11681.873866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11681.888800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11681.980136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11689.465641] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11693.738255] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11693.952858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11698.408593] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11698.690388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11702.757611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11702.764929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11702.777875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11702.782519] Lustre: Skipped 1 previous similar message [11706.860326] Lustre: server umount lustre-OST0000 complete [11712.998582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11713.006264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11713.036625] Lustre: Skipped 2 previous similar messages [11713.059466] Lustre: Skipped 1 previous similar message [11716.972224] Lustre: server umount lustre-MDT0000 complete [11720.796947] LustreError: 403243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847988 with bad export cookie 8167309406366762438 [11720.805418] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11720.821960] LustreError: 403243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11721.173245] Lustre: server umount lustre-MDT0001 complete [11729.757290] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11731.437632] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:26:36 (1787847996) [11737.953895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11738.370281] LustreError: 407679:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11738.396861] LustreError: 407679:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11738.480582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11743.204439] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11743.728675] LustreError: 407680:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11748.835978] LustreError: 407679:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11750.701306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11751.031859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11755.406355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11762.157831] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11766.581698] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11773.576460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.876810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11780.201963] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11787.408687] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11790.534174] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11793.223463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11799.521977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11799.531686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11799.532541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11799.565734] LustreError: Skipped 1 previous similar message [11804.646436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11804.652667] Lustre: Skipped 1 previous similar message [11805.677919] Lustre: server umount lustre-OST0000 complete [11809.769646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11809.771113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11809.787907] Lustre: Skipped 2 previous similar messages [11809.808701] Lustre: Skipped 2 previous similar messages [11815.375598] Lustre: server umount lustre-MDT0000 complete [11819.107112] LustreError: 407660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848086 with bad export cookie 8167309406366763327 [11819.116854] LustreError: 407660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11819.125551] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11819.552584] Lustre: server umount lustre-MDT0001 complete [11830.685287] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11833.197644] Key type lgssc unregistered [11833.473685] LNet: 411407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11833.484668] LNetError: 411407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11833.509201] LNet: Removed LNI 192.168.203.122@tcp [11834.445567] Key type .llcrypt unregistered [11834.447787] Key type ._llcrypt unregistered [11857.159619] Key type ._llcrypt registered [11857.160927] Key type .llcrypt registered [11857.248316] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11857.509502] Key type .llcrypt unregistered [11857.511296] Key type ._llcrypt unregistered [11873.271905] Key type ._llcrypt registered [11873.274414] Key type .llcrypt registered [11873.374435] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11874.162297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11874.262341] alg: No test for adler32 (adler32-zlib) [11875.365444] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [11875.576966] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [11877.264244] Key type lgssc registered [11878.435300] Lustre: Echo OBD driver; http://www.lustre.org/ [11886.488994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11886.511415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11887.816457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11891.728539] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11897.323989] LustreError: 413623:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11897.353696] LustreError: 413623:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11899.727134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11900.105897] LustreError: 413622:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.128233] LustreError: 413622:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11900.141955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11905.506737] LustreError: 413623:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11905.520699] LustreError: 413623:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11906.334142] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11910.645578] LustreError: 413622:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.662097] LustreError: 413622:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11912.709527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11914.214688] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11917.799508] LustreError: 414883:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.835253] LustreError: 414883:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11921.183863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.705146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11926.297144] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11937.256345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11937.258448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11937.281593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11937.816712] Lustre: server umount lustre-OST0000 complete [11941.983566] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11941.993675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11942.023556] Lustre: Skipped 1 previous similar message [11942.037681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11942.045614] Lustre: Skipped 1 previous similar message [11942.266840] Lustre: server umount lustre-MDT0000 complete [11942.375207] LustreError: 413622:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.398987] LustreError: 413622:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11945.609671] LustreError: 413603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848213 with bad export cookie 446913220790025008 [11945.613460] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11945.627702] LustreError: 413603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11952.111524] Lustre: server umount lustre-MDT0001 complete [11960.819312] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11963.520577] Key type lgssc unregistered [11963.823828] LNet: 416774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11963.837247] LNetError: 416774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11963.859463] LNet: Removed LNI 192.168.203.122@tcp [11964.642146] Key type .llcrypt unregistered [11964.645747] Key type ._llcrypt unregistered [11987.946556] Key type ._llcrypt registered [11987.949341] Key type .llcrypt registered [11988.074609] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11988.987670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11989.008438] alg: No test for adler32 (adler32-zlib) [11990.035849] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [11990.293145] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [11992.073157] Key type lgssc registered [11993.611113] Lustre: Echo OBD driver; http://www.lustre.org/ [12002.279995] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12002.318981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12017.952467] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12023.072453] LustreError: 418164:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [12049.826356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12055.902393] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12063.121967] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12068.931170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12069.330633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12069.414388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12073.863215] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12074.471238] LustreError: 418190:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12074.500571] LustreError: 418190:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12079.586279] LustreError: 418189:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12081.414085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12081.806181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12086.066591] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12092.894375] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12098.940570] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12107.237907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12107.259043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12107.269848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12108.499756] Lustre: server umount lustre-OST0000 complete [12112.358540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12112.366072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12112.372575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12115.938287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12115.944150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12121.058059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12121.069918] Lustre: Skipped 2 previous similar messages [12123.231861] Lustre: server umount lustre-MDT0000 complete [12126.180301] LustreError: 419207:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12126.213914] LustreError: 419207:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12127.749977] LustreError: 418170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848395 with bad export cookie 4549642713513249175 [12127.752974] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12127.770597] LustreError: 418170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12128.290293] Lustre: server umount lustre-MDT0001 complete [12139.252123] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12141.227860] Key type lgssc unregistered [12141.430562] LNet: 421737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12141.438289] LNetError: 421737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12141.455282] LNet: Removed LNI 192.168.203.122@tcp [12142.381074] Key type .llcrypt unregistered [12142.386343] Key type ._llcrypt unregistered [12167.825305] Key type ._llcrypt registered [12167.829752] Key type .llcrypt registered [12167.953118] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12168.945737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12169.140706] alg: No test for adler32 (adler32-zlib) [12170.228716] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [12170.551739] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [12172.279751] Key type lgssc registered [12173.531834] Lustre: Echo OBD driver; http://www.lustre.org/ [12180.787290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12180.820482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.407267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12187.123345] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12192.737568] LustreError: 423155:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.753764] LustreError: 423155:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12196.654769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12197.105397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12201.594752] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12208.195517] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12213.769727] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12220.562859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12220.890255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12226.614940] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12228.087776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12233.004630] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12238.309252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12238.314818] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12238.319862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12238.347162] Lustre: Skipped 2 previous similar messages [12239.722641] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [12239.727544] Lustre: Skipped 2 previous similar messages [12242.600345] Lustre: server umount lustre-MDT0000 complete [12243.431397] LustreError: 423883:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12243.452307] LustreError: 423883:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12244.983697] LustreError: 423135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.22@tcp arrived at 1787848512 with bad export cookie 14514763879397151422 [12245.005120] LustreError: 423135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12247.088303] LustreError: 425736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848514 with bad export cookie 14514763879397150526 [12247.101132] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12247.102199] LustreError: 425736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12247.524797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12247.536152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12247.553853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12248.545663] LustreError: 423155:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12248.560857] LustreError: 423155:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12252.578024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12252.578806] LustreError: 423156:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.584369] Lustre: Skipped 1 previous similar message [12252.610885] LustreError: 423156:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12253.297389] Lustre: server umount lustre-OST0000 complete [12256.675291] LustreError: 424397:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.692756] LustreError: 424397:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12260.102569] Lustre: server umount lustre-MDT0001 complete [12270.002420] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12272.319828] Key type lgssc unregistered [12272.592898] LNet: 426844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12272.602690] LNetError: 426844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12272.616350] LNet: Removed LNI 192.168.203.122@tcp [12273.364707] Key type .llcrypt unregistered [12273.368900] Key type ._llcrypt unregistered [12295.884221] Key type ._llcrypt registered [12295.887219] Key type .llcrypt registered [12296.038546] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12297.673176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12297.741189] alg: No test for adler32 (adler32-zlib) [12298.938738] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [12299.244773] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [12300.952598] Key type lgssc registered [12301.977491] Lustre: Echo OBD driver; http://www.lustre.org/ [12309.360476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12309.396561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12310.911757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12315.271845] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12321.254365] LustreError: 428262:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12321.280130] LustreError: 428262:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12324.060675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12324.369150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12328.777692] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12336.202698] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12342.190528] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12349.614176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12349.863253] LustreError: 430083:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12350.224323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12355.356747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12357.913155] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12366.572316] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12370.914231] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12370.942464] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12373.895968] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [12375.521946] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12375.529682] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12375.550506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12376.979265] Lustre: server umount lustre-MDT0001 complete [12378.953375] LustreError: 428257:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.157668] LustreError: 428990:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.182655] LustreError: 428990:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12386.280343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12386.284068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12386.286345] LustreError: 428263:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12386.303269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12386.348816] Lustre: Skipped 2 previous similar messages [12390.296755] Lustre: server umount lustre-OST0000 complete [12394.324689] Lustre: server umount lustre-MDT0000 complete [12407.372102] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12410.139563] Key type lgssc unregistered [12410.403129] LNet: 431952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12410.412578] LNetError: 431952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12410.442076] LNet: Removed LNI 192.168.203.122@tcp [12411.247690] Key type .llcrypt unregistered [12411.249753] Key type ._llcrypt unregistered [12433.944695] Key type ._llcrypt registered [12433.947854] Key type .llcrypt registered [12434.058124] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [12448.721555] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12450.039498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12450.071404] alg: No test for adler32 (adler32-zlib) [12451.133706] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [12451.399329] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [12453.048211] Key type lgssc registered [12454.423521] Lustre: Echo OBD driver; http://www.lustre.org/ [12489.834896] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12507.672900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12507.693880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12509.082901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12509.133691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12509.263432] Lustre: lustre-MDT0000: new disk, initializing [12509.368799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12509.405135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12514.445322] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12531.754772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12532.019867] Lustre: lustre-OST0000: new disk, initializing [12532.026309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12532.032730] Lustre: Skipped 1 previous similar message [12532.042789] Lustre: 436985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12532.149883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12538.453686] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12541.593564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12541.608407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12541.656465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12548.732620] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12565.489900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12565.749754] Lustre: 435967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12565.874480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12566.032591] Lustre: lustre-MDT0001: new disk, initializing [12566.274562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12566.346939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12566.362243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12571.628069] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12583.140811] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12583.353584] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12588.752749] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12588.963309] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12596.080090] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [12596.195585] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12596.199914] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12596.225295] Lustre: Skipped 1 previous similar message [12599.779486] Lustre: server umount lustre-MDT0001 complete [12606.437808] LustreError: 435976:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12606.461906] LustreError: 435976:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12611.556419] LustreError: 435980:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12611.573096] LustreError: 435980:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12616.682931] LustreError: 435976:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12616.701929] LustreError: 435976:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12618.727061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12618.738610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12618.754498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12618.761446] Lustre: Skipped 2 previous similar messages [12621.794900] LustreError: 435979:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12621.795939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12623.348570] Lustre: server umount lustre-OST0000 complete [12626.921047] LustreError: 435980:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12633.684276] Lustre: server umount lustre-MDT0000 complete [12646.353385] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12648.753954] Key type lgssc unregistered [12648.975756] LNet: 440736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12648.985584] LNetError: 440736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12650.023326] LNet: Removed LNI 192.168.203.122@tcp [12650.782775] Key type .llcrypt unregistered [12650.788829] Key type ._llcrypt unregistered [12698.750347] Key type ._llcrypt registered [12698.754502] Key type .llcrypt registered [12698.960634] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12699.540492] Key type .llcrypt unregistered [12699.547985] Key type ._llcrypt unregistered [12719.203222] Key type ._llcrypt registered [12719.204813] Key type .llcrypt registered [12719.297936] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12720.434634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12720.567484] alg: No test for adler32 (adler32-zlib) [12721.603491] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [12721.848363] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [12723.544193] Key type lgssc registered [12724.763935] Lustre: Echo OBD driver; http://www.lustre.org/ [12753.278226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12753.303743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12754.744333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12754.808545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12754.948454] Lustre: lustre-MDT0000: new disk, initializing [12755.080984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12755.118603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12761.147359] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12773.692553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12773.868593] Lustre: 444117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12773.964975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12773.971716] Lustre: Skipped 1 previous similar message [12774.096458] Lustre: lustre-MDT0001: new disk, initializing [12774.193309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12774.231799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12774.245293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12778.682756] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12788.113072] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12794.707115] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12801.117763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12801.385670] Lustre: lustre-OST0000: new disk, initializing [12801.390972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12801.397760] Lustre: 446092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12801.476615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12807.799762] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12810.316118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12810.325496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12810.453759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12817.783227] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12825.572634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12825.580920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12825.594704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12826.083074] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12826.118183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12826.120994] Lustre: Skipped 2 previous similar messages [12830.750387] Lustre: server umount lustre-MDT0000 complete [12834.480322] LustreError: 444918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849102 with bad export cookie 2991404600664595076 [12834.487795] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12834.493918] LustreError: 444918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12834.841637] Lustre: server umount lustre-MDT0001 complete [12847.466961] Lustre: server umount lustre-OST0000 complete [12857.260798] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:45:22 (1787849122) [12864.701726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12865.237353] LustreError: 448119:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12865.274015] LustreError: 448119:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12865.321288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12870.056795] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12870.625996] LustreError: 448120:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12874.728566] LustreError: 448119:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12879.503934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12880.016079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12884.183544] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12890.809452] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12895.545581] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12914.544139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12914.740336] Lustre: Found index 0 for lustre-OST0000, updating log [12914.955730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12922.329364] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12931.619201] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12935.483217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12945.889150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12945.909411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12945.923337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12945.929405] Lustre: Skipped 2 previous similar messages [12948.260990] Lustre: server umount lustre-OST0000 complete [12964.420474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12964.426816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12964.437654] Lustre: Skipped 1 previous similar message [12964.462113] Lustre: Skipped 4 previous similar messages [12969.444178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12969.448909] Lustre: Skipped 1 previous similar message [12970.846157] Lustre: server umount lustre-MDT0000 complete [12972.531931] LustreError: 448119:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12972.550254] LustreError: 448119:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12974.442386] LustreError: 448100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849242 with bad export cookie 2991404600664596476 [12974.443616] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12974.457581] LustreError: 448100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12974.852338] Lustre: server umount lustre-MDT0001 complete [12982.810361] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:47:28 (1787849248) [12985.057275] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp [13602.576412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13603.190365] LustreError: 454529:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13603.203704] LustreError: 454529:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13603.264484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13608.434650] LustreError: 454530:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13608.678691] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13612.530215] LustreError: 454529:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13618.586258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13618.658807] LustreError: 454530:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13618.683312] LustreError: 454530:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13618.924606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13623.702947] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13631.496667] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13637.918464] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13658.741681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13659.179928] Lustre: Found index 0 for lustre-OST0000, updating log [13659.574808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13663.968744] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp [13666.575996] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13674.035550] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13676.228420] Lustre: server umount lustre-OST0000 complete [13680.046242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13680.048157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13680.072041] Lustre: Skipped 1 previous similar message [13680.091752] Lustre: Skipped 1 previous similar message [13681.127640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13681.139113] Lustre: Skipped 1 previous similar message [13683.173745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13683.185284] Lustre: Skipped 3 previous similar messages [13686.443024] Lustre: server umount lustre-MDT0000 complete [13687.289777] LustreError: 454524:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13687.319811] LustreError: 454524:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13690.147225] LustreError: 454508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849957 with bad export cookie 2991404600664597554 [13690.152391] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13690.165327] LustreError: 454508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13690.642548] Lustre: server umount lustre-MDT0001 complete [13722.760383] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [13723.363607] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp [13735.685813] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [13775.805995] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:00:40 (1787850040) [13794.851318] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [13797.857848] Key type lgssc unregistered [13798.138336] LNet: 461164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13798.145643] LNetError: 461164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13920.039210] LNet: Removed LNI 192.168.203.122@tcp [13920.942121] Key type .llcrypt unregistered [13920.947120] Key type ._llcrypt unregistered [13936.488814] Key type ._llcrypt registered [13936.491919] Key type .llcrypt registered [13936.594665] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [13937.575787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13937.704801] alg: No test for adler32 (adler32-zlib) [13938.827421] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [13939.092948] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [13940.728329] Key type lgssc registered [13941.903506] Lustre: Echo OBD driver; http://www.lustre.org/ [13945.927637] LNet: Added LNI 192.168.203.122@tcp100 [8/256/0/180] [13946.732093] LNet: Added LNI 192.168.203.122@tcp101 [8/256/0/180] [13947.679199] LNet: Added LNI 192.168.203.122@tcp102 [8/256/0/180] [13948.532804] LNet: Added LNI 192.168.203.122@tcp103 [8/256/0/180] [13949.380706] LNet: Added LNI 192.168.203.122@tcp104 [8/256/0/180] [13950.307832] LNet: Added LNI 192.168.203.122@tcp105 [8/256/0/180] [13951.107477] LNet: Added LNI 192.168.203.122@tcp106 [8/256/0/180] [13952.027553] LNet: Added LNI 192.168.203.122@tcp107 [8/256/0/180] [13952.819757] LNet: Added LNI 192.168.203.122@tcp108 [8/256/0/180] [13953.591859] LNet: Added LNI 192.168.203.122@tcp109 [8/256/0/180] [13954.414804] LNet: Added LNI 192.168.203.122@tcp110 [8/256/0/180] [13955.273422] LNet: Added LNI 192.168.203.122@tcp111 [8/256/0/180] [13956.161211] LNet: Added LNI 192.168.203.122@tcp112 [8/256/0/180] [13956.961724] LNet: Added LNI 192.168.203.122@tcp113 [8/256/0/180] [13957.872744] LNet: Added LNI 192.168.203.122@tcp114 [8/256/0/180] [13958.831457] LNet: Added LNI 192.168.203.122@tcp115 [8/256/0/180] [13959.696785] LNet: Added LNI 192.168.203.122@tcp116 [8/256/0/180] [13960.594425] LNet: Added LNI 192.168.203.122@tcp117 [8/256/0/180] [13961.576214] LNet: Added LNI 192.168.203.122@tcp118 [8/256/0/180] [13962.381445] LNet: Added LNI 192.168.203.122@tcp119 [8/256/0/180] [13963.311981] LNet: Added LNI 192.168.203.122@tcp120 [8/256/0/180] [13964.115184] LNet: Added LNI 192.168.203.122@tcp121 [8/256/0/180] [13964.858683] LNet: Added LNI 192.168.203.122@tcp122 [8/256/0/180] [13965.541235] LNet: Added LNI 192.168.203.122@tcp123 [8/256/0/180] [13966.282608] LNet: Added LNI 192.168.203.122@tcp124 [8/256/0/180] [13966.990653] LNet: Added LNI 192.168.203.122@tcp125 [8/256/0/180] [13967.707460] LNet: Added LNI 192.168.203.122@tcp126 [8/256/0/180] [13968.759897] LNet: Added LNI 192.168.203.122@tcp127 [8/256/0/180] [13969.639329] LNet: Added LNI 192.168.203.122@tcp128 [8/256/0/180] [13970.558496] LNet: Added LNI 192.168.203.122@tcp129 [8/256/0/180] [13971.418584] LNet: Added LNI 192.168.203.122@tcp130 [8/256/0/180] [13972.422647] LNet: Added LNI 192.168.203.122@tcp131 [8/256/0/180] [13973.314321] LNet: Added LNI 192.168.203.122@tcp132 [8/256/0/180] [13974.113804] LNet: Added LNI 192.168.203.122@tcp133 [8/256/0/180] [13974.910406] LNet: Added LNI 192.168.203.122@tcp134 [8/256/0/180] [13975.668822] LNet: Added LNI 192.168.203.122@tcp135 [8/256/0/180] [13982.943078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13982.995353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13984.152706] Lustre: 464199:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [13984.223494] LNetError: 461658:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13984.240608] LNetError: Protocol error connecting to 192.168.203.122@tcp119 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13984.258243] LNetError: Skipped 2 previous similar messages [13984.382279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13984.409965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13984.500687] Lustre: lustre-MDT0000: new disk, initializing [13984.606963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13984.623560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13985.184380] LNet: 8 local NIs in recovery (showing 8): 192.168.203.122@tcp124, 192.168.203.122@tcp125, 192.168.203.122@tcp129, 192.168.203.122@tcp130, 192.168.203.122@tcp127, 192.168.203.122@tcp128, 192.168.203.122@tcp119, 192.168.203.122@tcp131 [13985.187749] LNetError: 464231:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [13985.239280] LNetError: 464231:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [13985.264517] LNetError: Protocol error connecting to 192.168.203.122@tcp105 at host 192.168.203.122:988: is it running a compatible version of Lustre? [13985.287646] LNetError: Skipped 3 previous similar messages [13989.230145] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14001.085716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14001.419736] Lustre: 464217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14001.538952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14001.554474] Lustre: Skipped 1 previous similar message [14001.690503] Lustre: lustre-MDT0001: new disk, initializing [14001.733525] LNetError: 461657:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [14001.765270] LNetError: Protocol error connecting to 192.168.203.122@tcp132 at host 192.168.203.122:988: is it running a compatible version of Lustre? [14001.769091] Lustre: 465018:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [14001.805303] Lustre: 465018:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14002.027850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14002.077792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14002.088830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14004.705448] LNetError: 464226:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [14004.723681] LNetError: 464226:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14004.736380] LNetError: Protocol error connecting to 192.168.203.122@tcp132 at host 192.168.203.122:988: is it running a compatible version of Lustre? [14004.751027] LNetError: Skipped 7 previous similar messages [14007.281124] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14008.803374] LNetError: 464219:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [14008.812292] LNetError: Protocol error connecting to 192.168.203.122@tcp132 at host 192.168.203.122:988: is it running a compatible version of Lustre? [14009.939741] Lustre: 465383:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [14009.948018] Lustre: 465383:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14016.929786] LNetError: 464222:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [14016.960480] LNetError: 464222:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14016.982201] LNetError: Protocol error connecting to 192.168.203.122@tcp123 at host 192.168.203.122:988: is it running a compatible version of Lustre? [14017.014274] LNetError: Skipped 1 previous similar message [14017.926474] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14024.993414] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14032.469265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14032.706357] Lustre: lustre-OST0000: new disk, initializing [14032.709886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14032.713291] Lustre: 466192:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14032.770715] Lustre: 466197:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [14032.777541] Lustre: 466197:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14032.843708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14033.322033] LNetError: 464229:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [14033.344964] LNetError: 464229:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [14033.362099] LNetError: Protocol error connecting to 192.168.203.122@tcp134 at host 192.168.203.122:988: is it running a compatible version of Lustre? [14033.387911] LNetError: Skipped 11 previous similar messages [14040.380169] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14041.624297] Lustre: 466617:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [14041.634434] Lustre: 466617:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14041.742392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14041.758061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14041.978603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14044.704194] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp132 [14050.012730] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14062.569911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14062.580680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14062.601800] Lustre: Skipped 1 previous similar message [14062.631682] Lustre: Skipped 1 previous similar message [14065.721863] Lustre: server umount lustre-OST0000 complete [14071.203822] LNetError: 464219:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [14071.222837] LNetError: 464219:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [14071.234698] LNetError: Protocol error connecting to 192.168.203.122@tcp134 at host 192.168.203.122:988: is it running a compatible version of Lustre? [14071.255377] LNetError: Skipped 18 previous similar messages [14072.801853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14072.832802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14072.837028] Lustre: Skipped 1 previous similar message [14072.879580] Lustre: Skipped 1 previous similar message [14076.552332] Lustre: server umount lustre-MDT0000 complete [14080.654104] LustreError: 464207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850348 with bad export cookie 4163413326097163517 [14080.669377] LustreError: 464207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14080.669787] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14081.154807] Lustre: server umount lustre-MDT0001 complete [14091.949632] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [14095.477680] Key type lgssc unregistered [14095.996084] LNet: 468109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14096.011384] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14097.056155] LNet: Removed LNI 192.168.203.122@tcp [14097.058118] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14097.065219] LNet: Removed LNI 192.168.203.122@tcp100 [14098.080167] LNet: Removed LNI 192.168.203.122@tcp101 [14098.107145] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14098.138531] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14098.154750] LNet: Removed LNI 192.168.203.122@tcp102 [14099.232746] LNet: Removed LNI 192.168.203.122@tcp103 [14100.256807] LNet: Removed LNI 192.168.203.122@tcp104 [14100.265244] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14100.275044] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14100.282259] LNet: Removed LNI 192.168.203.122@tcp105 [14101.344186] LNet: Removed LNI 192.168.203.122@tcp106 [14102.368690] LNet: Removed LNI 192.168.203.122@tcp107 [14103.392171] LNet: Removed LNI 192.168.203.122@tcp108 [14104.416484] LNet: Removed LNI 192.168.203.122@tcp109 [14104.419290] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14104.437158] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14104.447452] LNet: Removed LNI 192.168.203.122@tcp110 [14105.504175] LNet: Removed LNI 192.168.203.122@tcp111 [14106.528603] LNet: Removed LNI 192.168.203.122@tcp112 [14107.552228] LNet: Removed LNI 192.168.203.122@tcp113 [14108.576158] LNet: Removed LNI 192.168.203.122@tcp114 [14109.600161] LNet: Removed LNI 192.168.203.122@tcp115 [14110.624168] LNet: Removed LNI 192.168.203.122@tcp116 [14111.648279] LNet: Removed LNI 192.168.203.122@tcp117 [14112.672200] LNet: Removed LNI 192.168.203.122@tcp118 [14112.675348] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14112.689729] LNetError: 468109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14112.701245] LNet: Removed LNI 192.168.203.122@tcp119 [14113.761633] LNet: Removed LNI 192.168.203.122@tcp120 [14114.785516] LNet: Removed LNI 192.168.203.122@tcp121 [14115.808178] LNet: Removed LNI 192.168.203.122@tcp122 [14116.832168] LNet: Removed LNI 192.168.203.122@tcp123 [14117.857157] LNet: Removed LNI 192.168.203.122@tcp124 [14118.880188] LNet: Removed LNI 192.168.203.122@tcp125 [14119.904484] LNet: Removed LNI 192.168.203.122@tcp126 [14120.928160] LNet: Removed LNI 192.168.203.122@tcp127 [14121.952182] LNet: Removed LNI 192.168.203.122@tcp128 [14122.976170] LNet: Removed LNI 192.168.203.122@tcp129 [14124.000150] LNet: Removed LNI 192.168.203.122@tcp130 [14125.025444] LNet: Removed LNI 192.168.203.122@tcp131 [14125.034498] LNet: Removed LNI 192.168.203.122@tcp132 [14126.112334] LNet: Removed LNI 192.168.203.122@tcp133 [14127.136194] LNet: Removed LNI 192.168.203.122@tcp134 [14128.169117] LNet: Removed LNI 192.168.203.122@tcp135 [14128.859972] Key type .llcrypt unregistered [14128.865814] Key type ._llcrypt unregistered [14149.947506] Key type ._llcrypt registered [14149.953576] Key type .llcrypt registered [14150.095711] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [14166.117363] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14167.642913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14167.690529] alg: No test for adler32 (adler32-zlib) [14168.825980] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [14169.079422] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [14170.768156] Key type lgssc registered [14172.014866] Lustre: Echo OBD driver; http://www.lustre.org/ [14206.163862] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:07:51 (1787850471) [14223.506668] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [14226.272761] Key type lgssc unregistered [14226.625481] LNet: 472208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14226.634505] LNetError: 472208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14226.671224] LNet: Removed LNI 192.168.203.122@tcp [14227.549509] Key type .llcrypt unregistered [14227.555420] Key type ._llcrypt unregistered [14243.453542] Key type ._llcrypt registered [14243.456304] Key type .llcrypt registered [14243.555279] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14244.689548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14244.875692] alg: No test for adler32 (adler32-zlib) [14245.965961] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [14246.225253] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [14247.920105] Key type lgssc registered [14249.097430] Lustre: Echo OBD driver; http://www.lustre.org/ [14259.341528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14259.368125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14260.637694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14260.737396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14260.848644] Lustre: lustre-MDT0000: new disk, initializing [14260.967574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14260.992269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14266.419330] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14277.230386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14277.446034] Lustre: 473422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14277.519361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14277.523876] Lustre: Skipped 1 previous similar message [14277.608602] Lustre: lustre-MDT0001: new disk, initializing [14277.729438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14277.799311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14277.812981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14282.623448] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14293.023819] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14300.453359] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14310.429778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14311.043940] Lustre: lustre-OST0000: new disk, initializing [14311.049081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14311.058161] Lustre: 475390:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14311.224889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14318.673951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14318.692585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14318.854895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14319.564223] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14330.391561] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14334.698687] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing lnet_if_list [14341.196355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14341.228672] LNet: Added LNI 192.168.203.122@tcp10 [8/256/0/180] [14346.577416] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14346.609772] LNet: Added LNI 192.168.203.122@tcp11 [8/256/0/180] [14351.001930] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14351.034635] LNet: Added LNI 192.168.203.122@tcp12 [8/256/0/180] [14355.312757] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14355.362108] LNet: Added LNI 192.168.203.122@tcp13 [8/256/0/180] [14359.667273] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14359.710189] LNet: Added LNI 192.168.203.122@tcp14 [8/256/0/180] [14364.892161] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14364.918219] LNet: Added LNI 192.168.203.122@tcp15 [8/256/0/180] [14369.680271] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14369.738185] LNet: Added LNI 192.168.203.122@tcp16 [8/256/0/180] [14374.473635] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14374.517322] LNet: Added LNI 192.168.203.122@tcp17 [8/256/0/180] [14379.460691] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14379.481869] LNet: Added LNI 192.168.203.122@tcp18 [8/256/0/180] [14384.101574] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14384.142657] LNet: Added LNI 192.168.203.122@tcp19 [8/256/0/180] [14390.459932] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14390.493504] LNet: Added LNI 192.168.203.122@tcp20 [8/256/0/180] [14397.226910] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14397.262608] LNet: Added LNI 192.168.203.122@tcp21 [8/256/0/180] [14404.034731] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14404.090903] LNet: Added LNI 192.168.203.122@tcp22 [8/256/0/180] [14409.977611] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14410.050585] LNet: Added LNI 192.168.203.122@tcp23 [8/256/0/180] [14416.563086] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14416.597028] LNet: Added LNI 192.168.203.122@tcp24 [8/256/0/180] [14423.994378] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14424.042834] LNet: Added LNI 192.168.203.122@tcp25 [8/256/0/180] [14431.781723] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14431.821938] LNet: Added LNI 192.168.203.122@tcp26 [8/256/0/180] [14439.686373] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14439.766459] LNet: Added LNI 192.168.203.122@tcp27 [8/256/0/180] [14448.297351] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14448.418334] LNet: Added LNI 192.168.203.122@tcp28 [8/256/0/180] [14456.169469] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14456.210302] LNet: Added LNI 192.168.203.122@tcp29 [8/256/0/180] [14463.430206] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14463.460234] LNet: Added LNI 192.168.203.122@tcp30 [8/256/0/180] [14470.114392] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14470.146809] LNet: Added LNI 192.168.203.122@tcp31 [8/256/0/180] [14476.229154] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14476.264592] LNet: Added LNI 192.168.203.122@tcp32 [8/256/0/180] [14483.127788] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14483.171723] LNet: Added LNI 192.168.203.122@tcp33 [8/256/0/180] [14489.742424] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14489.763261] LNet: Added LNI 192.168.203.122@tcp34 [8/256/0/180] [14497.447964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14497.508401] LNet: Added LNI 192.168.203.122@tcp35 [8/256/0/180] [14504.266400] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14504.309898] LNet: Added LNI 192.168.203.122@tcp36 [8/256/0/180] [14512.121127] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14512.157549] LNet: Added LNI 192.168.203.122@tcp37 [8/256/0/180] [14519.412404] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14519.445657] LNet: Added LNI 192.168.203.122@tcp38 [8/256/0/180] [14526.551343] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14526.598801] LNet: Added LNI 192.168.203.122@tcp39 [8/256/0/180] [14533.731727] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14533.784684] LNet: Added LNI 192.168.203.122@tcp40 [8/256/0/180] [14541.022984] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14541.061106] LNet: Added LNI 192.168.203.122@tcp41 [8/256/0/180] [14549.487573] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14549.514939] LNet: Added LNI 192.168.203.122@tcp42 [8/256/0/180] [14558.485845] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14558.585074] LNet: Added LNI 192.168.203.122@tcp43 [8/256/0/180] [14572.677740] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14572.760871] LNet: Added LNI 192.168.203.122@tcp44 [8/256/0/180] [14583.243794] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14583.293648] LNet: Added LNI 192.168.203.122@tcp45 [8/256/0/180] [14592.688601] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14592.724552] LNet: Added LNI 192.168.203.122@tcp46 [8/256/0/180] [14603.882099] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14603.971093] LNet: Added LNI 192.168.203.122@tcp47 [8/256/0/180] [14614.005443] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14614.039861] LNet: Added LNI 192.168.203.122@tcp48 [8/256/0/180] [14623.499992] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14623.559229] LNet: Added LNI 192.168.203.122@tcp49 [8/256/0/180] [14632.089914] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14632.135641] LNet: Added LNI 192.168.203.122@tcp50 [8/256/0/180] [14651.368527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14651.377489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14651.400101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14651.881581] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14651.906814] Lustre: Skipped 1 previous similar message [14651.925990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14651.928550] Lustre: Skipped 1 previous similar message [14656.406980] Lustre: server umount lustre-MDT0000 complete [14659.983904] LustreError: 478429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850927 with bad export cookie 6557010611784649937 [14659.995280] LustreError: 478429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14660.002199] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14660.360933] Lustre: server umount lustre-MDT0001 complete [14677.411317] Lustre: 472696:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850929/real 1787850929] req@00000000d999a28a x1874697139696256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14677.453992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14677.874970] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14679.525308] Lustre: 472694:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850931/real 1787850931] req@000000002c2af0d2 x1874697139696384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850947 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14682.592584] Lustre: 472693:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850934/real 1787850934] req@00000000624fd7c9 x1874697139696512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14685.602298] Lustre: 472696:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850937/real 1787850937] req@000000006a8b80d7 x1874697139696768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14687.973558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14691.808134] Lustre: 472695:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850943/real 1787850943] req@000000009d246f9d x1874697139696896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850959 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14705.492729] LustreError: 475384:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14705.515528] LustreError: 475384:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14705.564209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14706.594158] LustreError: 475385:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14706.600049] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14708.657830] LustreError: 475384:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14710.237955] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14713.834683] LustreError: 475385:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14718.708364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14718.951513] LustreError: 475384:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14719.209680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14723.302369] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14723.305324] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14731.484517] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14738.281248] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14759.406091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14759.411368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14759.423116] Lustre: Skipped 1 previous similar message [14759.437797] Lustre: Skipped 1 previous similar message [14764.522632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14764.531069] Lustre: Skipped 1 previous similar message [14769.636913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14769.641169] Lustre: Skipped 1 previous similar message [14772.192141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14772.338925] Lustre: server umount lustre-OST0000 complete [14779.879412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14779.898896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14779.909226] Lustre: Skipped 1 previous similar message [14782.300627] Lustre: server umount lustre-MDT0000 complete [14784.998789] LustreError: 499637:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14785.021190] LustreError: 499637:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14786.466655] LustreError: 473415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851054 with bad export cookie 6557010611784652541 [14786.476798] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14786.492893] LustreError: 473415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14786.828895] Lustre: server umount lustre-MDT0001 complete [14797.406311] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [14800.252652] Key type lgssc unregistered [14800.600787] LNet: 502447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14800.613112] LNetError: 502447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14800.621704] LNet: Removed LNI 192.168.203.122@tcp [14800.624617] LNet: Removed LNI 192.168.203.122@tcp10 [14800.627482] LNet: Removed LNI 192.168.203.122@tcp11 [14800.631465] LNet: Removed LNI 192.168.203.122@tcp12 [14800.634736] LNet: Removed LNI 192.168.203.122@tcp13 [14800.637667] LNet: Removed LNI 192.168.203.122@tcp14 [14800.641921] LNet: Removed LNI 192.168.203.122@tcp15 [14800.644456] LNet: Removed LNI 192.168.203.122@tcp16 [14800.651305] LNet: Removed LNI 192.168.203.122@tcp17 [14800.655768] LNet: Removed LNI 192.168.203.122@tcp18 [14800.667357] LNet: Removed LNI 192.168.203.122@tcp19 [14800.670955] LNet: Removed LNI 192.168.203.122@tcp20 [14800.673241] LNet: Removed LNI 192.168.203.122@tcp21 [14800.678323] LNet: Removed LNI 192.168.203.122@tcp22 [14800.681842] LNet: Removed LNI 192.168.203.122@tcp23 [14800.685551] LNet: Removed LNI 192.168.203.122@tcp24 [14800.688658] LNet: Removed LNI 192.168.203.122@tcp25 [14800.693536] LNet: Removed LNI 192.168.203.122@tcp26 [14800.696667] LNet: Removed LNI 192.168.203.122@tcp27 [14800.700156] LNet: Removed LNI 192.168.203.122@tcp28 [14800.703470] LNet: Removed LNI 192.168.203.122@tcp29 [14800.706342] LNet: Removed LNI 192.168.203.122@tcp30 [14800.710394] LNet: Removed LNI 192.168.203.122@tcp31 [14800.716501] LNet: Removed LNI 192.168.203.122@tcp32 [14800.719506] LNet: Removed LNI 192.168.203.122@tcp33 [14800.722804] LNet: Removed LNI 192.168.203.122@tcp34 [14800.729775] LNet: Removed LNI 192.168.203.122@tcp35 [14800.732745] LNet: Removed LNI 192.168.203.122@tcp36 [14800.739579] LNet: Removed LNI 192.168.203.122@tcp37 [14800.745119] LNet: Removed LNI 192.168.203.122@tcp38 [14800.747320] LNet: Removed LNI 192.168.203.122@tcp39 [14800.750752] LNet: Removed LNI 192.168.203.122@tcp40 [14800.755489] LNet: Removed LNI 192.168.203.122@tcp41 [14800.758824] LNet: Removed LNI 192.168.203.122@tcp42 [14800.762473] LNet: Removed LNI 192.168.203.122@tcp43 [14800.765306] LNet: Removed LNI 192.168.203.122@tcp44 [14800.767739] LNet: Removed LNI 192.168.203.122@tcp45 [14800.769917] LNet: Removed LNI 192.168.203.122@tcp46 [14800.772631] LNet: Removed LNI 192.168.203.122@tcp47 [14800.775853] LNet: Removed LNI 192.168.203.122@tcp48 [14800.779149] LNet: Removed LNI 192.168.203.122@tcp49 [14800.788856] LNet: Removed LNI 192.168.203.122@tcp50 [14801.712907] Key type .llcrypt unregistered [14801.718161] Key type ._llcrypt unregistered [14824.103605] Key type ._llcrypt registered [14824.105514] Key type .llcrypt registered [14824.221392] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [14840.425129] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14841.662989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14841.851669] alg: No test for adler32 (adler32-zlib) [14842.961897] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [14843.290891] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [14844.992367] Key type lgssc registered [14846.161753] Lustre: Echo OBD driver; http://www.lustre.org/ [14881.961468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14881.988446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14883.286616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14883.314366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14883.387258] Lustre: lustre-MDT0000: new disk, initializing [14883.528515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14883.553128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14888.511383] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14899.721173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14899.854806] Lustre: 505644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14899.906956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14899.912071] Lustre: Skipped 1 previous similar message [14899.988078] Lustre: lustre-MDT0001: new disk, initializing [14900.116385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14900.168608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14900.184044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14905.224442] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14915.472123] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14920.810718] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14928.369799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14928.572535] Lustre: *** cfs_fail_loc=90f, val=0*** [14928.647722] Lustre: lustre-OST0000: new disk, initializing [14928.651967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14928.664448] Lustre: 507660:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14928.783258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14935.598720] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [14937.173054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14937.185109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14937.327579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14944.367481] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14952.930298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14952.946168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14952.964971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14958.068776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14958.080939] Lustre: Skipped 2 previous similar messages [14963.177381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14963.190939] Lustre: Skipped 1 previous similar message [14966.240253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14966.459929] Lustre: server umount lustre-OST0000 complete [14971.874579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14971.885441] LustreError: Skipped 1 previous similar message [14971.889716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14971.909841] Lustre: Skipped 1 previous similar message [14971.925896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14973.413613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14977.992454] Lustre: server umount lustre-MDT0000 complete [14982.294586] LustreError: 505638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851249 with bad export cookie 14685101611625628941 [14982.295402] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14982.305091] LustreError: 505638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14982.781255] Lustre: server umount lustre-MDT0001 complete [14996.348540] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [15001.242350] Key type lgssc unregistered [15001.600631] LNet: 509573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15001.606126] LNetError: 509573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15001.630290] LNet: Removed LNI 192.168.203.122@tcp [15002.962246] Key type .llcrypt unregistered [15002.967265] Key type ._llcrypt unregistered [15026.310763] Key type ._llcrypt registered [15026.314794] Key type .llcrypt registered [15026.433118] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [15027.587940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15027.729644] alg: No test for adler32 (adler32-zlib) [15028.897731] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [15029.218773] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [15030.872181] Key type lgssc registered [15032.116847] Lustre: Echo OBD driver; http://www.lustre.org/ [15041.217113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15041.242633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15043.144534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15049.365988] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15053.287666] LustreError: 510846:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15053.329862] LustreError: 510846:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15058.413045] LustreError: 510845:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15061.912678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15062.598279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15068.794750] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15076.484872] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15081.778771] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15088.862324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15089.126500] LustreError: 512666:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15089.468740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15096.387633] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15103.639587] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15118.170947] loop0: detected capacity change from 0 to 204800000 [15118.445715] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15118.473381] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15118.493605] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15125.498897] loop0: detected capacity change from 0 to 204800000 [15125.713588] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15125.737982] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15125.752926] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15131.026829] loop0: detected capacity change from 0 to 204800000 [15136.686943] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15136.891865] Lustre: fs15246-OST0000: new disk, initializing [15136.896835] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15136.912683] Lustre: 514444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15136.992107] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15144.012248] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15154.874306] loop1: detected capacity change from 0 to 204800000 [15160.863190] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15160.991633] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15161.056940] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15161.138203] Lustre: fs15246-MDT0000: new disk, initializing [15161.332643] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15161.367928] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15161.492193] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15162.531376] Lustre: 515564:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006b2a426d x1874697960604160/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787851446 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15166.742430] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15171.629965] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15171.640984] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15171.669948] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15194.107466] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15194.120072] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15196.073737] Lustre: server umount fs15246-MDT0000 complete [15199.209440] LustreError: 510841:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15204.325800] LustreError: 510842:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15205.171368] Lustre: server umount fs15246-OST0000 complete [15219.686298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15219.690893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15219.698573] Lustre: Skipped 1 previous similar message [15219.707830] Lustre: Skipped 1 previous similar message [15224.177655] Lustre: server umount lustre-OST0000 complete [15229.925365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15229.951857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15229.958775] Lustre: Skipped 1 previous similar message [15234.061796] Lustre: server umount lustre-MDT0000 complete [15235.053606] LustreError: 510845:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15235.078264] LustreError: 510845:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15237.666859] LustreError: 510827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851505 with bad export cookie 7252672379561309687 [15237.668809] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15237.675013] LustreError: 510827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15238.093268] Lustre: server umount lustre-MDT0001 complete [15248.568759] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [15251.547207] Key type lgssc unregistered [15251.920930] LNet: 518663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15251.931631] LNetError: 518663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15251.954597] LNet: Removed LNI 192.168.203.122@tcp [15252.891690] Key type .llcrypt unregistered [15252.898110] Key type ._llcrypt unregistered [15272.825510] Key type ._llcrypt registered [15272.827552] Key type .llcrypt registered [15272.945766] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [15274.019417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15274.141871] alg: No test for adler32 (adler32-zlib) [15275.200173] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [15275.458956] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [15277.160779] Key type lgssc registered [15278.276807] Lustre: Echo OBD driver; http://www.lustre.org/ [15319.100898] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [15329.834720] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [15375.067401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15375.092867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15376.475217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15376.508213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15376.663859] Lustre: lustre-MDT0000: new disk, initializing [15376.828522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15376.862457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15381.390717] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15392.513029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15392.676446] Lustre: 523133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15392.753440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15392.762294] Lustre: Skipped 1 previous similar message [15392.847262] Lustre: lustre-MDT0001: new disk, initializing [15392.944026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15392.998147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15393.021833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15398.260394] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15409.355489] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15417.670463] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15425.803318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15426.207317] Lustre: lustre-OST0000: new disk, initializing [15426.211390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15426.217803] Lustre: 525101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15426.340396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15434.170325] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [15434.430486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15434.438266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15434.616221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15445.454975] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15451.837785] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15452.168845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15457.057901] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15457.267764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15460.322268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15460.333762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15460.343948] LustreError: Skipped 1 previous similar message [15460.374272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15465.694530] Lustre: server umount lustre-OST0000 complete [15475.175511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15475.180553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15475.189687] Lustre: Skipped 1 previous similar message [15475.195534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15475.208423] Lustre: Skipped 1 previous similar message [15477.308708] Lustre: server umount lustre-MDT0000 complete [15480.803808] LustreError: 523144:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15480.845956] LustreError: 523144:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [15482.742530] LustreError: 523126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851750 with bad export cookie 4709483216262606282 [15482.750193] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15482.754839] LustreError: 523126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15483.067799] Lustre: server umount lustre-MDT0001 complete [15513.842986] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [15530.251721] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [15565.457705] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15266 sec ========= 13:30:30 (1787851830) [15566.848732] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:30:32 (1787851832) === [15570.874657] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:30:35 (1787851835) === [15612.888867] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [15616.051709] Key type lgssc unregistered [15616.423347] LNet: 531773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15616.443647] LNetError: 531773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15616.469056] LNet: Removed LNI 192.168.203.122@tcp [15617.439205] Key type .llcrypt unregistered [15617.446190] Key type ._llcrypt unregistered