[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467641594 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003070] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011064] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.014053] *** VALIDATE selinux *** [ 0.021661] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025759] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] *** VALIDATE tmpfs *** [ 0.030198] *** VALIDATE proc *** [ 0.031251] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033259] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040185] debug: unmapping init [mem 0xffffffff99c59000-0xffffffff99c60fff] [ 0.042747] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044028] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051275] rcu: Hierarchical SRCU implementation. [ 0.053423] smp: Bringing up secondary CPUs ... [ 0.054551] x86: Booting SMP configuration: [ 0.055021] .... node #0, CPUs: #1 #2 #3 [ 0.058241] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.214527] node 0 deferred pages initialised in 150ms [ 0.218265] devtmpfs: initialized [ 0.220327] x86/mm: Memory block size: 128MB [ 0.223980] gcov: version magic: 0x41383552 [ 0.228152] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.232190] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.236296] pinctrl core: initialized pinctrl subsystem [ 0.238360] [ 0.240014] ************************************************************* [ 0.242025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.245021] ** ** [ 0.248020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251019] ** ** [ 0.253020] ** This means that this kernel is built to expose internal ** [ 0.256021] ** IOMMU data structures, which may compromise security on ** [ 0.259020] ** your system. ** [ 0.261015] ** ** [ 0.264020] ** If you see this message and you are not debugging the ** [ 0.267033] ** kernel, report this immediately to your vendor! ** [ 0.269016] ** ** [ 0.272025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.274162] ************************************************************* [ 0.277840] NET: Registered protocol family 16 [ 0.281575] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.284098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.288167] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.293130] cpuidle: using governor menu [ 0.295053] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.296676] PCI: Using configuration type 1 for base access [ 0.300154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.312143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.315034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.320032] cryptd: max_cpu_qlen set to 1000 [ 0.323288] ACPI: Added _OSI(Module Device) [ 0.324025] ACPI: Added _OSI(Processor Device) [ 0.326022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.328022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334352] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.343370] ACPI: Interpreter enabled [ 0.344073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.346022] ACPI: Using IOAPIC for interrupt routing [ 0.348199] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.352504] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.363277] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.367070] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.370040] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.374113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.380509] acpiphp: Slot [2] registered [ 0.382193] acpiphp: Slot [5] registered [ 0.383222] acpiphp: Slot [6] registered [ 0.386205] acpiphp: Slot [7] registered [ 0.387248] acpiphp: Slot [8] registered [ 0.389217] acpiphp: Slot [9] registered [ 0.391207] acpiphp: Slot [10] registered [ 0.393198] acpiphp: Slot [3] registered [ 0.395144] acpiphp: Slot [4] registered [ 0.396144] acpiphp: Slot [11] registered [ 0.398197] acpiphp: Slot [12] registered [ 0.400203] acpiphp: Slot [13] registered [ 0.401104] acpiphp: Slot [14] registered [ 0.403148] acpiphp: Slot [15] registered [ 0.405132] acpiphp: Slot [16] registered [ 0.407198] acpiphp: Slot [17] registered [ 0.408210] acpiphp: Slot [18] registered [ 0.410150] acpiphp: Slot [19] registered [ 0.412148] acpiphp: Slot [20] registered [ 0.414167] acpiphp: Slot [21] registered [ 0.415215] acpiphp: Slot [22] registered [ 0.417191] acpiphp: Slot [23] registered [ 0.418222] acpiphp: Slot [24] registered [ 0.420149] acpiphp: Slot [25] registered [ 0.422150] acpiphp: Slot [26] registered [ 0.424156] acpiphp: Slot [27] registered [ 0.427174] acpiphp: Slot [28] registered [ 0.428207] acpiphp: Slot [29] registered [ 0.430250] acpiphp: Slot [30] registered [ 0.432167] acpiphp: Slot [31] registered [ 0.434099] PCI host bridge to bus 0000:00 [ 0.436026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.438034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.441031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.444035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.447035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.450062] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.453239] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.456282] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.460501] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.471021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.477089] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.479025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.482032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.486064] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.489683] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.491912] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.496060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.499074] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.505023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.519270] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.526011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.531692] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.544023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.552018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.572020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.583547] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.594023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.604022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.625017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.637150] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.644000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.650022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.680023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.692608] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.707020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.713021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.733022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.744392] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.751018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.763022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.784022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.797468] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.806024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.815022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.837030] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.850747] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.854397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.856381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.859415] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.861215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.868048] iommu: Default domain type: Passthrough [ 0.869512] SCSI subsystem initialized [ 0.871167] ACPI: bus type USB registered [ 0.873126] usbcore: registered new interface driver usbfs [ 0.875078] usbcore: registered new interface driver hub [ 0.877086] usbcore: registered new device driver usb [ 0.878182] pps_core: LinuxPPS API ver. 1 registered [ 0.880011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.882054] PTP clock support registered [ 0.884210] EDAC MC: Ver: 3.0.0 [ 0.886135] PCI: Using ACPI for IRQ routing [ 0.887897] NetLabel: Initializing [ 0.889011] NetLabel: domain hash size = 128 [ 0.890011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.893101] NetLabel: unlabeled traffic allowed by default [ 0.895159] vgaarb: loaded [ 0.896299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.898017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.903307] clocksource: Switched to clocksource kvm-clock [ 1.017614] VFS: Disk quotas dquot_6.6.0 [ 1.019243] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.021545] *** VALIDATE ramfs *** [ 1.023227] *** VALIDATE hugetlbfs *** [ 1.025147] pnp: PnP ACPI init [ 1.027890] pnp: PnP ACPI: found 6 devices [ 1.045986] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.049140] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.052811] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.056658] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.060651] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.064363] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.068638] NET: Registered protocol family 2 [ 1.071871] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.077695] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.083387] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.089427] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.093718] TCP: Hash tables configured (established 65536 bind 65536) [ 1.097413] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.101405] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.105165] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.109186] NET: Registered protocol family 1 [ 1.112217] RPC: Registered named UNIX socket transport module. [ 1.115061] RPC: Registered udp transport module. [ 1.117150] RPC: Registered tcp transport module. [ 1.119155] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.121694] NET: Registered protocol family 44 [ 1.123955] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.126282] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.128473] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.130980] PCI: CLS 0 bytes, default 64 [ 1.133225] Unpacking initramfs... [ 2.587325] debug: unmapping init [mem 0xffff929f3cc54000-0xffff929f3ffbffff] [ 2.591639] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.594759] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.597470] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.117923] Initialise system trusted keyrings [ 3.119755] Key type blacklist registered [ 3.122816] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.131200] zbud: loaded [ 3.133908] *** VALIDATE nfs *** [ 3.135272] *** VALIDATE nfs4 *** [ 3.136849] pstore: using deflate compression [ 3.139528] Platform Keyring initialized [ 3.255398] NET: Registered protocol family 38 [ 3.257556] Key type asymmetric registered [ 3.259424] Asymmetric key parser 'x509' registered [ 3.261467] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.265157] io scheduler mq-deadline registered [ 3.267149] io scheduler kyber registered [ 3.269042] io scheduler bfq registered [ 3.271059] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.274398] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.277909] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.281854] ACPI: Power Button [PWRF] [ 3.287865] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.296151] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.311627] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.321372] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.344246] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.373036] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.402181] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.408831] Non-volatile memory driver v1.3 [ 3.410760] Linux agpgart interface v0.103 [ 3.446078] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.450450] vda: detected capacity change from 0 to 77115392 [ 3.464740] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.468322] vdb: detected capacity change from 0 to 1073741824 [ 3.485651] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.490378] vdc: detected capacity change from 0 to 2621440000 [ 3.509215] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.512692] vdd: detected capacity change from 0 to 2621440000 [ 3.530117] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.533235] vde: detected capacity change from 0 to 4294967296 [ 3.550173] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.553170] vdf: detected capacity change from 0 to 4294967296 [ 3.563747] libphy: Fixed MDIO Bus: probed [ 3.568700] usbcore: registered new interface driver usbserial_generic [ 3.571142] usbserial: USB Serial support registered for generic [ 3.573724] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.578548] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.580240] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.582622] mousedev: PS/2 mouse device common for all mice [ 3.586444] rtc_cmos 00:05: RTC can wake from S4 [ 3.590637] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.591469] rtc_cmos 00:05: registered as rtc0 [ 3.598043] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.602082] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.602437] intel_pstate: CPU model not supported [ 3.608744] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.609160] hid: raw HID events driver (C) Jiri Kosina [ 3.614293] usbcore: registered new interface driver usbhid [ 3.616577] usbhid: USB HID core driver [ 3.618399] drop_monitor: Initializing network drop monitor service [ 3.620863] Initializing XFRM netlink socket [ 3.622970] NET: Registered protocol family 10 [ 3.626247] Segment Routing with IPv6 [ 3.627668] NET: Registered protocol family 17 [ 3.630397] mpls_gso: MPLS GSO support [ 3.636420] RAS: Correctable Errors collector initialized. [ 3.638749] AVX version of gcm_enc/dec engaged. [ 3.640548] AES CTR mode by8 optimization enabled [ 3.730291] sched_clock: Marking stable (3730266108, 0)->(4678710734, -948444626) [ 3.743314] registered taskstats version 1 [ 3.747461] Loading compiled-in X.509 certificates [ 3.749760] zswap: loaded using pool lzo/zbud [ 3.779537] Key type big_key registered [ 3.794056] Key type encrypted registered [ 3.795904] ima: No TPM chip found, activating TPM-bypass! [ 3.798362] ima: Allocated hash algorithm: sha1 [ 3.800463] ima: No architecture policies found [ 3.802606] evm: Initialising EVM extended attributes: [ 3.805110] evm: security.selinux [ 3.806584] evm: security.ima [ 3.807783] evm: security.capability [ 3.809459] evm: HMAC attrs: 0x1 [ 3.813766] rtc_cmos 00:05: setting system clock to 2026-10-02 12:51:34 UTC (1790945494) [ 3.819643] debug: unmapping init [mem 0xffffffff9ac03000-0xffffffff9adfffff] [ 3.823284] debug: unmapping init [mem 0xffffffff99982000-0xffffffff99c58fff] [ 3.834092] Write protecting the kernel read-only data: 28672k [ 3.837626] debug: unmapping init [mem 0xffffffff98003000-0xffffffff981fffff] [ 3.840363] debug: unmapping init [mem 0xffffffff98914000-0xffffffff989fffff] [ 3.881991] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.892604] systemd[1]: Detected virtualization kvm. [ 3.894571] systemd[1]: Detected architecture x86-64. [ 3.896710] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.926271] systemd[1]: No hostname configured. [ 3.927794] systemd[1]: Set hostname to . [ 3.930173] random: systemd: uninitialized urandom read (16 bytes read) [ 3.932772] systemd[1]: Initializing machine ID from random generator. [ 4.072444] random: systemd: uninitialized urandom read (16 bytes read) [ 4.074870] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.079208] random: systemd: uninitialized urandom read (16 bytes read) [ 4.081946] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.087147] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.751553] device-mapper: uevent: version 1.0.3 [ 4.753941] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.509274] virtio_net virtio0 ens2: renamed from eth0 [ 5.536575] random: fast init done [ 5.606274] scsi host0: ata_piix [ 5.616341] scsi host1: ata_piix [ 5.618271] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.620890] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.331187] random: crng init done [ 10.337191] random: 7 urandom warning(s) missed due to ratelimiting [ 10.481675] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.936481] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.036244] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.933273] SELinux: Disabled at runtime. [ 16.067599] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.087383] systemd[1]: Detected virtualization kvm. [ 16.095655] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.451435] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.456178] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.464493] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.475398] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.481110] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.501573] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.522276] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ 17.945099] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.854723] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.149642] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.347642] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.537715] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.578624] EDAC sbridge: Ver: 1.1.2 [ 24.583501] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 25.248457] NFS: Registering the id_resolver key type [ 25.251138] Key type id_resolver registered [ 25.253225] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. [ 35.480017] hrtimer: interrupt took 5986907 ns Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg626-server login: [ 90.300239] libcfs: loading out-of-tree module taints kernel. [ 90.337847] Key type ._llcrypt registered [ 90.340926] Key type .llcrypt registered [ 90.515535] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 111.472693] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 114.185771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.223891] alg: No test for adler32 (adler32-zlib) [ 116.071732] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 117.511537] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 119.464267] Key type lgssc registered [ 121.563202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 138.042300] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 177.279674] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 192.015407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 192.079746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.475197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.534107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.665544] Lustre: lustre-MDT0000: new disk, initializing [ 193.772960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.799248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 198.715903] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.811509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.935955] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.970941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 213.978143] Lustre: Skipped 1 previous similar message [ 214.121264] Lustre: lustre-MDT0001: new disk, initializing [ 214.229702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.289571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 214.309121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.780688] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.019639] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.540429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.771456] Lustre: lustre-OST0000: new disk, initializing [ 233.776348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 233.790360] Lustre: 8445:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.852960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.224127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.239529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.279664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.247962] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.424143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.610616] Lustre: lustre-OST0001: new disk, initializing [ 254.615625] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.622364] Lustre: 9520:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.680893] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.160697] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.187752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.202339] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.300094] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 276.147896] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.136195] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 292.424986] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing check_logdir /tmp/testlogs/ [ 297.546552] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing yml_node [ 302.465485] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 305.202888] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 308.059098] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 310.223802] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 08:56:38 EDT 2026 [ 330.216218] Lustre: DEBUG MARKER: only running tests: 33-75 [ 332.072830] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 333.741532] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 352.736931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 352.750677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.795383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.786094] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.786978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.806175] Lustre: Skipped 1 previous similar message [ 354.813910] Lustre: Skipped 3 previous similar messages [ 356.963660] Lustre: server umount lustre-MDT0000 complete [ 360.943097] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790945851 with bad export cookie 182890331049633423 [ 360.943421] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.955276] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 365.025260] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.027829] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.057895] LustreError: 6516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 365.084321] Lustre: Skipped 1 previous similar message [ 365.115298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.130322] Lustre: Skipped 1 previous similar message [ 367.516163] Lustre: server umount lustre-MDT0001 complete [ 371.987360] Lustre: server umount lustre-OST0000 complete [ 375.972464] Lustre: server umount lustre-OST0001 complete [ 383.097478] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 393.830133] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 428.000425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.281117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.333647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.455856] Lustre: lustre-MDT0000: new disk, initializing [ 428.571864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.585838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 432.659915] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 444.548857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.715755] Lustre: 15120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 444.874875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 444.889022] Lustre: Skipped 1 previous similar message [ 445.006468] Lustre: lustre-MDT0001: new disk, initializing [ 445.138803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 445.202842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 445.217292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 450.033740] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 459.370636] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 465.632734] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 473.039727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 473.421180] Lustre: lustre-OST0000: new disk, initializing [ 473.425238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 473.435707] Lustre: 17093:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 473.542731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.524569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 475.534321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 475.723412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 479.800918] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 489.119638] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 493.525189] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 493.698226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 498.736633] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 498.958816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 501.219069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.221204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.225947] LustreError: Skipped 1 previous similar message [ 501.238560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.244615] Lustre: Skipped 2 previous similar messages [ 506.345188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.354818] Lustre: Skipped 1 previous similar message [ 507.163890] Lustre: server umount lustre-OST0000 complete [ 511.462535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.486075] Lustre: Skipped 1 previous similar message [ 516.578161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.581790] Lustre: Skipped 3 previous similar messages [ 525.280277] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 525.444058] Lustre: server umount lustre-MDT0000 complete [ 526.818993] LustreError: 16001:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.837859] LustreError: 16001:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 528.828094] LustreError: 15927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946019 with bad export cookie 182890331049634956 [ 528.830831] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.838738] LustreError: 15927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 529.107243] Lustre: server umount lustre-MDT0001 complete [ 533.102454] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:00:21 (1790946021) [ 535.512927] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 537.352943] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:00:25 (1790946025) [ 545.119427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.569911] LustreError: 19502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.665452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.581870] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 550.887787] LustreError: 19503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.013470] LustreError: 19502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.469106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.112335] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 572.076117] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 577.898899] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 584.956714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 590.988103] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 598.126908] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 606.192334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 606.203582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.235567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 606.239421] Lustre: Skipped 2 previous similar messages [ 611.658391] Lustre: server umount lustre-OST0000 complete [ 616.417634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 621.510909] Lustre: server umount lustre-MDT0000 complete [ 621.550852] LustreError: 19503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.568224] LustreError: 19503:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 625.470501] LustreError: 19482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946116 with bad export cookie 182890331049635838 [ 625.486560] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.905676] Lustre: server umount lustre-MDT0001 complete [ 636.500174] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 639.066708] Key type lgssc unregistered [ 639.322730] LNet: 23047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.335160] LNetError: 23047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.351496] LNet: Removed LNI 192.168.206.126@tcp [ 640.133729] Key type .llcrypt unregistered [ 640.136720] Key type ._llcrypt unregistered [ 661.777438] Key type ._llcrypt registered [ 661.780372] Key type .llcrypt registered [ 661.868255] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 662.788329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 662.890371] alg: No test for adler32 (adler32-zlib) [ 664.069837] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 664.312183] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 666.000688] Key type lgssc registered [ 667.337814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 675.094063] loop: module loaded [ 675.176755] loop0: detected capacity change from 0 to 204800000 [ 675.374971] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 675.398556] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 675.807725] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.514357] loop0: detected capacity change from 0 to 204800000 [ 681.670908] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.679538] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.703021] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.602853] loop0: detected capacity change from 0 to 204800000 [ 692.229892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 692.268933] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.491753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 693.539604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 693.645728] Lustre: lustre-MDT0000: new disk, initializing [ 693.708496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 693.723745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 697.715938] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 707.708662] loop1: detected capacity change from 0 to 204800000 [ 713.132958] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 713.411274] Lustre: lustre-OST07c6: new disk, initializing [ 713.416360] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 713.422865] Lustre: Skipped 1 previous similar message [ 713.429673] Lustre: 26253:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 713.495317] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 719.292591] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 719.934448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 719.942753] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 719.972802] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 723.908039] Lustre: 25238:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 723.917271] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 723.923015] Lustre: 25238:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 723.930293] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 723.939955] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 723.944875] Lustre: 25238:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 736.785988] Lustre: 25238:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 736.790690] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 736.799975] Lustre: 25238:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 736.804747] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 736.812732] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 736.821014] Lustre: 25238:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 741.553169] Lustre: 25237:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 741.570383] Lustre: 25237:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 741.582348] Lustre: 25237:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 741.587429] Lustre: 25237:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.592362] Lustre: 25237:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 741.599876] Lustre: 25237:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.604580] Lustre: 25237:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 741.609809] Lustre: 25237:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.614846] Lustre: 25237:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 741.620577] Lustre: 25237:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.627088] Lustre: 25237:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 741.631565] Lustre: 25237:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.786588] Lustre: 25239:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 744.798160] Lustre: 25239:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 744.810877] Lustre: 25239:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 744.819033] Lustre: 25239:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.830497] Lustre: 25239:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 744.842870] Lustre: 25239:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.851345] Lustre: 25239:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 744.861293] Lustre: 25239:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.866482] Lustre: 25239:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 744.872867] Lustre: 25239:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.885169] Lustre: 25239:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.892335] Lustre: 25239:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.310494] Lustre: 25238:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 749.317796] Lustre: 25238:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 749.326524] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 749.334159] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.344690] Lustre: 25238:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 749.352814] Lustre: 25238:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.361587] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 749.369187] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.375665] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 749.382180] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 749.391398] Lustre: 25238:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 749.400910] Lustre: 25238:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.561355] Lustre: 25239:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 760.574410] Lustre: 25239:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 760.583309] Lustre: 25239:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.592322] Lustre: 25239:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 760.605705] Lustre: 25239:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.614330] Lustre: 25239:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 760.626407] Lustre: 25239:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 760.637112] Lustre: 25239:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 760.644527] Lustre: 25239:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 760.650253] Lustre: 25239:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 760.657270] Lustre: 25239:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.663097] Lustre: 25239:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 812.007305] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.020653] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 812.028253] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 815.640646] Lustre: server umount lustre-OST07c6 complete [ 826.527282] Lustre: server umount lustre-MDT0000 complete [ 842.814195] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 845.698722] Key type lgssc unregistered [ 846.013161] LNet: 28806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.019505] LNetError: 28806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.041664] LNet: Removed LNI 192.168.206.126@tcp [ 847.014200] Key type .llcrypt unregistered [ 847.018522] Key type ._llcrypt unregistered [ 866.074544] Key type ._llcrypt registered [ 866.076411] Key type .llcrypt registered [ 866.161535] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 867.418188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 867.500137] alg: No test for adler32 (adler32-zlib) [ 868.586861] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 868.853739] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 870.689111] Key type lgssc registered [ 871.685297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 879.105308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 879.124174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.527086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.546051] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 890.853524] LustreError: 30093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 890.865443] LustreError: 30093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 892.972722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.358490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 897.292954] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 904.780185] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 910.283291] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 917.200965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 917.440480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.534760] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 925.628496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 932.523648] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 951.273853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.278342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.300583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.103343] Lustre: server umount lustre-OST0000 complete [ 961.520947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 961.527893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 961.546755] Lustre: Skipped 2 previous similar messages [ 961.563700] Lustre: Skipped 2 previous similar messages [ 966.625781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.637677] Lustre: Skipped 1 previous similar message [ 971.745115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 971.763438] Lustre: Skipped 1 previous similar message [ 975.328728] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 975.849975] Lustre: server umount lustre-MDT0000 complete [ 976.865517] LustreError: 32587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.900679] LustreError: 32587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 980.461183] LustreError: 31939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946471 with bad export cookie 6222847667886063010 [ 980.483448] LustreError: 31939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 980.488022] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.927652] Lustre: server umount lustre-MDT0001 complete [ 992.859079] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 995.624763] Key type lgssc unregistered [ 995.907801] LNet: 33832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.917784] LNetError: 33832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.948886] LNet: Removed LNI 192.168.206.126@tcp [ 996.814378] Key type .llcrypt unregistered [ 996.819696] Key type ._llcrypt unregistered [ 1014.391703] Key type ._llcrypt registered [ 1014.395543] Key type .llcrypt registered [ 1014.498809] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 1015.644909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.730641] alg: No test for adler32 (adler32-zlib) [ 1016.815643] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1017.015377] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 1018.664227] Key type lgssc registered [ 1019.842385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1028.020746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1028.056859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.670870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.290356] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1039.847713] LustreError: 35116:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.859360] LustreError: 35116:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1042.604288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.987582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1047.120187] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1054.512441] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1059.591204] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1065.560488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1065.824099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.754387] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1075.072714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1077.332939] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1088.483685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1088.494676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1088.515989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1090.532446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1090.537632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1090.548874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.791531] Lustre: server umount lustre-OST0000 complete [ 1099.236841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1099.247869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.262163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1099.276458] Lustre: Skipped 1 previous similar message [ 1103.907158] Lustre: server umount lustre-MDT0000 complete [ 1105.892038] LustreError: 35115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.915805] LustreError: 35115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1107.958335] LustreError: 35095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946598 with bad export cookie 3819509769350552664 [ 1107.969375] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.483356] Lustre: server umount lustre-MDT0001 complete [ 1124.201766] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 1128.333198] Key type lgssc unregistered [ 1128.743697] LNet: 38647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.758057] LNetError: 38647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.779383] LNet: Removed LNI 192.168.206.126@tcp [ 1129.855113] Key type .llcrypt unregistered [ 1129.857885] Key type ._llcrypt unregistered [ 1153.083954] Key type ._llcrypt registered [ 1153.090273] Key type .llcrypt registered [ 1153.185903] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 1153.858608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1153.969086] alg: No test for adler32 (adler32-zlib) [ 1155.185818] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1155.486094] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 1157.168762] Key type lgssc registered [ 1158.235707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1165.856935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1165.871417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.277703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.322734] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1176.547764] LustreError: 40080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.561164] LustreError: 40080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1179.057621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.358961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1183.342598] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1190.142285] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1195.255939] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1202.586148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1202.996887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.310670] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1212.218982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1217.943439] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1225.702463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.708868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.723255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.755673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.757563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.781379] Lustre: Skipped 3 previous similar messages [ 1230.927230] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.26@tcp (stopping) [ 1231.185114] Lustre: server umount lustre-MDT0000 complete [ 1232.867953] LustreError: 40079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.885550] LustreError: 40079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1234.984572] LustreError: 40059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946725 with bad export cookie 8716232587347588475 [ 1234.986908] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.000292] LustreError: 40059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1236.056092] LustreError: 40075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.086639] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.26@tcp (stopping) [ 1237.986388] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.015536] Lustre: Skipped 1 previous similar message [ 1240.034192] LustreError: 41628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.054932] LustreError: 41628:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1241.160694] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.26@tcp (stopping) [ 1241.173906] Lustre: Skipped 2 previous similar messages [ 1241.388102] Lustre: server umount lustre-MDT0001 complete [ 1262.853337] Lustre: server umount lustre-OST0000 complete [ 1280.116485] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 1283.111567] Key type lgssc unregistered [ 1283.477530] LNet: 43940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.486044] LNetError: 43940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.518520] LNet: Removed LNI 192.168.206.126@tcp [ 1284.573910] Key type .llcrypt unregistered [ 1284.579575] Key type ._llcrypt unregistered [ 1306.842610] Key type ._llcrypt registered [ 1306.845323] Key type .llcrypt registered [ 1306.937971] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 1307.919909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1307.986768] alg: No test for adler32 (adler32-zlib) [ 1309.083251] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1309.363722] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 1311.040202] Key type lgssc registered [ 1312.146681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1319.253714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1319.277019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.818585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1324.984769] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1331.175869] LustreError: 45365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1331.195826] LustreError: 45365:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1332.763655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.079371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1337.324960] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1344.539218] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1349.667839] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1355.992218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1356.337695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.984170] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1365.571690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1368.683726] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1375.327161] Lustre: lustre-OST0000: Not available for connect from 192.168.206.26@tcp (stopping) [ 1375.719131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1375.731634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.742631] Lustre: Skipped 1 previous similar message [ 1380.224144] Lustre: server umount lustre-OST0000 complete [ 1399.786457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1399.804551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.816659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1399.821480] Lustre: Skipped 2 previous similar messages [ 1401.321730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.324576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.895058] Lustre: server umount lustre-MDT0000 complete [ 1406.440840] LustreError: 45365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.472456] LustreError: 45365:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1408.157728] LustreError: 45344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946898 with bad export cookie 15849861053286860915 [ 1408.175128] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1414.543438] Lustre: server umount lustre-MDT0001 complete [ 1424.700558] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 1427.604084] Key type lgssc unregistered [ 1427.939902] LNet: 49075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.946945] LNetError: 49075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.958832] LNet: Removed LNI 192.168.206.126@tcp [ 1428.738668] Key type .llcrypt unregistered [ 1428.742323] Key type ._llcrypt unregistered [ 1449.012456] Key type ._llcrypt registered [ 1449.022074] Key type .llcrypt registered [ 1449.169622] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 1450.095212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1450.177102] alg: No test for adler32 (adler32-zlib) [ 1451.403092] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1451.697453] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 1453.408894] Key type lgssc registered [ 1454.540563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1462.840610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1462.882794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.421794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.774620] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1475.049769] LustreError: 50486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.070891] LustreError: 50486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1477.419721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.787262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1482.117979] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1490.380415] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1496.744734] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1503.486160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1503.807494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.530206] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1509.936606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1516.246944] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1530.224875] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1533.781353] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1545.621711] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-10-02 9h17m13s [ 1547.001388] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1549.283777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1549.293729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.311645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1550.375043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1550.385304] Lustre: Skipped 2 previous similar messages [ 1553.510270] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.26@tcp (stopping) [ 1553.519299] Lustre: Skipped 2 previous similar messages [ 1555.344071] Lustre: server umount lustre-MDT0000 complete [ 1556.464789] LustreError: 50480:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.495561] LustreError: 50480:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1558.678374] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1560.656704] LustreError: 52990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.651428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.771380] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1565.050399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.115413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1566.079045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1568.711459] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1572.117329] Lustre: DEBUG MARKER: Wait for df (36922) ... [ 1573.624624] Lustre: DEBUG MARKER: done [ 1579.493922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1579.506071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.514051] Lustre: Skipped 2 previous similar messages [ 1579.519748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1579.522425] Lustre: Skipped 1 previous similar message [ 1580.517613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1580.528604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.526410] Lustre: server umount lustre-OST0000 complete [ 1591.782434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1591.796062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.798522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.814365] Lustre: Skipped 1 previous similar message [ 1591.823031] Lustre: Skipped 4 previous similar messages [ 1593.239706] Lustre: server umount lustre-MDT0000 complete [ 1597.055609] LustreError: 53141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947087 with bad export cookie 3128565362689207615 [ 1597.062973] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1597.074893] LustreError: 53141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1597.587721] Lustre: server umount lustre-MDT0001 complete [ 1608.358526] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 1611.835121] Key type lgssc unregistered [ 1612.261143] LNet: 55434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.273470] LNetError: 55434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.303680] LNet: Removed LNI 192.168.206.126@tcp [ 1613.186641] Key type .llcrypt unregistered [ 1613.188485] Key type ._llcrypt unregistered [ 1651.817740] Key type ._llcrypt registered [ 1651.820085] Key type .llcrypt registered [ 1651.920821] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 1652.779893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1652.807125] alg: No test for adler32 (adler32-zlib) [ 1653.954721] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1654.168549] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 1655.832298] Key type lgssc registered [ 1656.974948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1664.232569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1664.255487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.627449] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1665.650692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1665.851026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.088527] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1677.893570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.032718] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1678.061780] Lustre: 57658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1678.361332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1682.238482] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1689.944070] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1696.205315] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1703.994771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1704.166457] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1704.328575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.772393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1711.068135] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1718.745313] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1726.115563] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-10-02 9h20m14s [ 1727.886586] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1735.112042] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1737.514879] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1737.559061] LustreError: 57664:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1757.813396] Lustre: lustre-MDT0000: Client d97a1807-c595-46a4-a8e8-bc0a8b26107f (at 192.168.206.26@tcp) reconnecting [ 1777.560133] LustreError: 57664:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1779.908809] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1782.414032] Lustre: DEBUG MARKER: done [ 1792.482228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1792.491628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.523543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1793.001213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1793.014556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1793.047316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1795.661623] Lustre: server umount lustre-OST0000 complete [ 1801.188416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1801.199753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.213578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.235750] Lustre: Skipped 1 previous similar message [ 1805.479531] Lustre: server umount lustre-MDT0000 complete [ 1809.497780] LustreError: 59515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947300 with bad export cookie 12553261210081040852 [ 1809.502445] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1809.509070] LustreError: 59515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1809.893872] Lustre: server umount lustre-MDT0001 complete [ 1819.316894] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 1822.213531] Key type lgssc unregistered [ 1822.479479] LNet: 61985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.484510] LNetError: 61985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.500388] LNet: Removed LNI 192.168.206.126@tcp [ 1823.210961] Key type .llcrypt unregistered [ 1823.213276] Key type ._llcrypt unregistered [ 1868.322236] Key type ._llcrypt registered [ 1868.326530] Key type .llcrypt registered [ 1868.459887] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 1869.745263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.774911] alg: No test for adler32 (adler32-zlib) [ 1870.820148] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1871.020330] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 1872.673096] Key type lgssc registered [ 1873.620770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1876.370657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1876.399311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.666712] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1877.688127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1877.872077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1878.789935] Lustre: Failing over lustre-MDT0000 [ 1879.017422] Lustre: server umount lustre-MDT0000 complete [ 1885.188744] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:22:53 (1790947373) [ 1891.259841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.539649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.983448] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1902.415255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.523806] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1902.548668] Lustre: 64805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1902.808511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1907.047850] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1913.783688] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1918.755356] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1924.888701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1925.092844] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1925.278630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.513540] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1933.419433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1937.779503] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1943.298894] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1949.153988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1949.163527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.169087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.178610] Lustre: Skipped 1 previous similar message [ 1949.193277] Lustre: Skipped 2 previous similar messages [ 1954.278383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.284595] Lustre: Skipped 2 previous similar messages [ 1954.805689] Lustre: server umount lustre-MDT0000 complete [ 1958.422969] LustreError: 64798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947449 with bad export cookie 14270411663254062521 [ 1958.425970] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1958.446339] LustreError: 64798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1958.838413] Lustre: server umount lustre-MDT0001 complete [ 1961.800243] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1965.802260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1973.317909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.712153] Lustre: 63712:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947450/real 1790947450] req@00000000c3ea7688 x1877944420045568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790947466 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1975.746816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1975.763137] Lustre: Skipped 1 previous similar message [ 1979.872904] Lustre: 63710:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947454/real 1790947454] req@0000000066c87753 x1877944420045696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790947470 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1980.896284] Lustre: 63709:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947455/real 1790947455] req@00000000bcec8b91 x1877944420045824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790947471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1984.324984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.465765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1985.313162] Lustre: 63712:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947459/real 1790947459] req@0000000062ed662c x1877944420046080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790947475 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1988.380254] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1989.410282] Lustre: 63711:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947464/real 1790947464] req@0000000084597d4a x1877944420046208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790947480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1991.649396] LustreError: 68573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.700068] LustreError: 68573:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1996.773139] LustreError: 66654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.781789] LustreError: 66654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1996.821886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.152179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2002.428932] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2002.429265] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2004.531071] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2008.512977] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2012.892313] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2019.818368] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.822390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.830979] Lustre: Skipped 1 previous similar message [ 2019.848072] Lustre: Skipped 2 previous similar messages [ 2020.881278] Lustre: server umount lustre-MDT0000 complete [ 2024.326382] LustreError: 64796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947515 with bad export cookie 14270411663254064796 [ 2024.330898] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.333949] LustreError: 64796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2024.696508] Lustre: server umount lustre-MDT0001 complete [ 2030.162305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2037.737732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.312284] Lustre: 63711:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947515/real 1790947515] req@000000008f6d131b x1877944420071552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790947531 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2041.327378] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.334375] Lustre: Skipped 1 previous similar message [ 2049.570693] LustreError: 63708:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008fcb8d4e x1877944420072448/t0(0) o250->MGC192.168.206.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2049.911879] LustreError: 69309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.990532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.067041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2052.011836] LustreError: 66654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.038598] LustreError: 66654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2054.653368] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2057.185979] LustreError: 69309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.220776] LustreError: 69309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2060.269444] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2064.371248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.111069] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2070.006190] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2076.350068] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.200891] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2090.358415] Lustre: server umount lustre-MDT0000 complete [ 2090.470064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2090.478485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.488964] LustreError: 66655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.492292] Lustre: Skipped 2 previous similar messages [ 2090.530038] LustreError: 66655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2093.194992] LustreError: 64797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947583 with bad export cookie 14270411663254066175 [ 2093.204235] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2093.414793] Lustre: server umount lustre-MDT0001 complete [ 2098.322902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2100.955518] Lustre: DEBUG MARKER: files compared the same [ 2112.826639] Lustre: server umount lustre-OST0000 complete [ 2123.217383] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 2125.204215] Key type lgssc unregistered [ 2125.413957] LNet: 74526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.420241] LNetError: 74526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.431592] LNet: Removed LNI 192.168.206.126@tcp [ 2126.146165] Key type .llcrypt unregistered [ 2126.148651] Key type ._llcrypt unregistered [ 2144.069574] Key type ._llcrypt registered [ 2144.075499] Key type .llcrypt registered [ 2144.258456] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2146.073235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2146.206893] alg: No test for adler32 (adler32-zlib) [ 2147.324485] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2147.654330] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 2149.432189] Key type lgssc registered [ 2151.299564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2161.233114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2161.272826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.978989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.284379] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug +malloc all [ 2172.387036] LustreError: 75937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.413480] LustreError: 75937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2177.512402] LustreError: 75936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.801903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.046072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.490886] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug +malloc all [ 2190.641246] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2195.845881] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2201.886373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2202.093301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2207.024489] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug +malloc all [ 2208.247561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2213.577284] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2223.139380] Lustre: server umount lustre-OST0000 complete [ 2228.705203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.707892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.716123] Lustre: Skipped 1 previous similar message [ 2228.730331] Lustre: Skipped 1 previous similar message [ 2233.826084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.831398] Lustre: Skipped 1 previous similar message [ 2237.028137] Lustre: server umount lustre-MDT0000 complete [ 2238.945193] LustreError: 75937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.966700] LustreError: 75937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2239.968188] Lustre: 75236:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947714/real 1790947714] req@00000000209c1f94 x1877944710489728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790947730 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2239.968602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.002016] Lustre: 75236:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2240.412922] LustreError: 75918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947731 with bad export cookie 5142984456202633881 [ 2240.430594] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.932328] Lustre: server umount lustre-MDT0001 complete [ 2250.872401] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 2253.196755] Key type lgssc unregistered [ 2253.419869] LNet: 79467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.426947] LNetError: 79467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.441505] LNet: Removed LNI 192.168.206.126@tcp [ 2254.229146] Key type .llcrypt unregistered [ 2254.232690] Key type ._llcrypt unregistered [ 2273.671425] Key type ._llcrypt registered [ 2273.674143] Key type .llcrypt registered [ 2273.783937] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2274.465923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.590368] alg: No test for adler32 (adler32-zlib) [ 2275.615745] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2275.833361] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 2277.480263] Key type lgssc registered [ 2278.430785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2283.874451] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2283.894822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.424779] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2304.544408] LustreError: 80850:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.126@tcp: failed processing log, type 4: rc = -110 [ 2331.235638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.403911] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2346.562763] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2354.110701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.212831] Lustre: *** cfs_fail_loc=706, val=0*** [ 2354.658443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.727971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2359.075315] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2359.784012] LustreError: 80876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.805658] LustreError: 80876:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2364.899977] LustreError: 80875:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.925065] LustreError: 80876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.942619] LustreError: 80876:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2366.541505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.855437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2370.501664] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2377.703682] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2382.957885] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2385.382644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2385.392678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.401369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2387.430339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2387.432132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2387.447461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.479424] Lustre: Skipped 1 previous similar message [ 2392.549926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.566759] Lustre: Skipped 1 previous similar message [ 2397.666976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.675695] Lustre: Skipped 1 previous similar message [ 2399.200368] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2399.374430] Lustre: server umount lustre-OST0000 complete [ 2402.785483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.788568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2402.808985] Lustre: Skipped 1 previous similar message [ 2402.827290] Lustre: Skipped 1 previous similar message [ 2408.381995] Lustre: server umount lustre-MDT0000 complete [ 2412.195609] LustreError: 80858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947902 with bad export cookie 6388901786959018241 [ 2412.209815] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2412.214639] LustreError: 80858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2412.705144] Lustre: server umount lustre-MDT0001 complete [ 2422.269543] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 2424.999864] Key type lgssc unregistered [ 2425.322650] LNet: 84468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.334498] LNetError: 84468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.351059] LNet: Removed LNI 192.168.206.126@tcp [ 2426.158930] Key type .llcrypt unregistered [ 2426.163368] Key type ._llcrypt unregistered [ 2448.039111] Key type ._llcrypt registered [ 2448.041779] Key type .llcrypt registered [ 2448.131215] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2449.266224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2449.289080] alg: No test for adler32 (adler32-zlib) [ 2450.266508] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2450.448961] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 2452.104204] Key type lgssc registered [ 2453.230677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2459.842640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2459.863190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.760707] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2470.975234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.365630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.625192] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2476.514502] LustreError: 86587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.539174] LustreError: 86587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2480.613593] LustreError: 86591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.630965] LustreError: 86591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2482.236943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2482.469690] LustreError: 86587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.498816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.594241] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2487.786647] LustreError: 86590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.809517] LustreError: 86590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2492.920212] LustreError: 86587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.946743] LustreError: 86587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2495.659898] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2495.688876] Lustre: MGS: Client a928e790-c62d-4b3e-8a5b-9c02d99edb98 (at 0@lo) reconnecting [ 2495.713832] LustreError: 86601:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2495.895252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.958444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2499.817543] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2516.451338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2516.458632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.475872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2517.482491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2517.484364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.485148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2518.894274] Lustre: server umount lustre-OST0000 complete [ 2522.597319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.601024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2522.603464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2522.603472] Lustre: Skipped 1 previous similar message [ 2522.617166] Lustre: Skipped 1 previous similar message [ 2527.715912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.725557] Lustre: Skipped 2 previous similar messages [ 2528.438187] Lustre: server umount lustre-MDT0000 complete [ 2532.295510] Lustre: server umount lustre-MDT0001 complete [ 2536.024694] Lustre: server umount MGS complete [ 2547.205703] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:33:55 (1790948035) [ 2572.808707] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 2581.196653] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2612.234075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.378533] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2624.509967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.732596] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2624.782812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2624.868840] Lustre: lustre-MDT0001: new disk, initializing [ 2624.970075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2625.000831] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2625.030378] Lustre: 93753:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2625.045794] LustreError: 93753:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2629.192608] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2638.838320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2639.073583] Lustre: lustre-OST0000: new disk, initializing [ 2639.077496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2639.086892] Lustre: 94520:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2639.172386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.049926] Lustre: 94813:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000964f43c x1877945028221696/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790948152 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2646.076443] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2655.437985] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2661.453914] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2661.465330] Lustre: MGS: Client e31e7b17-e481-4c30-88be-9961a12839b6 (at 0@lo) reconnecting [ 2661.511671] Lustre: 93020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2661.532770] LustreError: 93738:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2661.541767] LustreError: 93738:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2661.552131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2661.601302] Lustre: lustre-MDT0000: new disk, initializing [ 2661.717518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.740884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2661.762323] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2662.833100] Lustre: 95591:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000076fda881 x1877945028228480/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790948169 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2666.130978] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2667.939921] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2671.980270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2671.988089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2672.023528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2697.190159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.203533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2697.221129] Lustre: Skipped 1 previous similar message [ 2701.159941] Lustre: server umount lustre-OST0000 complete [ 2707.432290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2707.432404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2707.443292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2707.453625] Lustre: Skipped 1 previous similar message [ 2707.468388] Lustre: Skipped 2 previous similar messages [ 2710.863716] Lustre: server umount lustre-MDT0000 complete [ 2712.557258] LustreError: 93722:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.583191] LustreError: 93722:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2720.837639] Lustre: server umount lustre-MDT0001 complete [ 2724.516273] Lustre: server umount MGS complete [ 2735.766900] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:37:04 (1790948224) [ 2752.614887] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 2755.244077] Key type lgssc unregistered [ 2755.559399] LNet: 98085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2755.578283] LNetError: 98085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2755.592905] LNet: Removed LNI 192.168.206.126@tcp [ 2756.375779] Key type .llcrypt unregistered [ 2756.380742] Key type ._llcrypt unregistered [ 2772.547830] Key type ._llcrypt registered [ 2772.553517] Key type .llcrypt registered [ 2772.662925] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2773.639591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2773.691313] alg: No test for adler32 (adler32-zlib) [ 2774.735954] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2774.998330] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 2776.697921] Key type lgssc registered [ 2778.187901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2785.383563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2785.412832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.941274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.425117] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2796.515131] LustreError: 99322:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.541993] LustreError: 99322:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2799.720060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.136169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2804.298273] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2811.022294] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2816.505446] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2823.469492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2823.780938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.879269] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2831.982528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2836.881730] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2845.754497] Lustre: server umount lustre-OST0000 complete [ 2851.303136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2851.320467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.339199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.841026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.843873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.878376] Lustre: Skipped 1 previous similar message [ 2857.955876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.962659] Lustre: Skipped 1 previous similar message [ 2863.072142] Lustre: 98623:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948337/real 1790948337] req@00000000a07cdc21 x1877945367946240/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790948353 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2863.076636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.100521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2863.122572] Lustre: Skipped 1 previous similar message [ 2864.096365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2864.097363] Lustre: 98622:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948338/real 1790948338] req@0000000099124561 x1877945367946880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790948354 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2864.407599] Lustre: server umount lustre-MDT0000 complete [ 2868.194175] LustreError: 101815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.244397] LustreError: 101815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2868.273022] LustreError: 99303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948358 with bad export cookie 1660685105389083123 [ 2868.274709] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2868.291452] LustreError: 99303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2868.669697] Lustre: server umount lustre-MDT0001 complete [ 2878.677452] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 2881.279757] Key type lgssc unregistered [ 2881.589270] LNet: 102856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2881.597758] LNetError: 102856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2881.610834] LNet: Removed LNI 192.168.206.126@tcp [ 2882.289919] Key type .llcrypt unregistered [ 2882.293222] Key type ._llcrypt unregistered [ 2899.545285] Key type ._llcrypt registered [ 2899.548976] Key type .llcrypt registered [ 2899.642251] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2900.934040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2900.988666] alg: No test for adler32 (adler32-zlib) [ 2902.118712] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2902.316360] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 2903.976531] Key type lgssc registered [ 2905.288140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2910.891086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2910.897120] LustreError: 103956:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2910.918782] LustreError: 103957:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2910.926852] LustreError: 103956:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 2910.939172] LustreError: 103956:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2910.948694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.955572] LustreError: 103956:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2912.606306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2920.404992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.855255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2925.524676] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2931.849563] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing lsmod [ 2934.843486] LustreError: 105163:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2934.897148] LustreError: 105166:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2934.907392] LustreError: 105163:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4943 [ 2934.919933] LustreError: 105163:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2934.929792] LustreError: 105163:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2934.944470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2935.462243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.521587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2941.410115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2941.420499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.429927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.947611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.951209] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.960524] Lustre: Skipped 1 previous similar message [ 2944.434986] Lustre: server umount lustre-MDT0000 complete [ 2946.089771] LustreError: 103995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.125902] LustreError: 103995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2948.480092] LustreError: 103970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948439 with bad export cookie 4083972386752194149 [ 2948.480826] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2948.497720] LustreError: 103970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2950.240411] LustreError: 104426:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000020c9826e x1877945501111040/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2950.456388] Lustre: server umount lustre-MDT0001 complete [ 2963.714244] Lustre: server umount lustre-OST0000 complete [ 2971.423975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.870931] LustreError: 106228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.953424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.970344] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2977.252592] LustreError: 106229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.351338] LustreError: 106228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.516349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.007715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2989.688315] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2997.136062] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3002.651853] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3009.357905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3009.507299] LustreError: 108049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.716853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.844540] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3017.217494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 3025.278699] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3032.545379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.551863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.559153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.581754] Lustre: Skipped 1 previous similar message [ 3032.595300] Lustre: Skipped 3 previous similar messages [ 3034.472489] Lustre: server umount lustre-OST0000 complete [ 3041.249397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3041.257466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.271733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3045.224351] Lustre: server umount lustre-MDT0000 complete [ 3047.905573] LustreError: 107232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.931209] LustreError: 107232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3049.550328] LustreError: 106209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948540 with bad export cookie 4083972386752195080 [ 3049.561540] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3049.567859] LustreError: 106209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3050.256660] Lustre: server umount lustre-MDT0001 complete [ 3061.962713] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 3064.739093] Key type lgssc unregistered [ 3065.159734] LNet: 109765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3065.173927] LNetError: 109765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3065.190968] LNet: Removed LNI 192.168.206.126@tcp [ 3066.001516] Key type .llcrypt unregistered [ 3066.003938] Key type ._llcrypt unregistered [ 3091.244991] Key type ._llcrypt registered [ 3091.247593] Key type .llcrypt registered [ 3091.340646] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 3092.359686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3092.503212] alg: No test for adler32 (adler32-zlib) [ 3093.654758] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3093.879655] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 3095.560262] Key type lgssc registered [ 3096.688102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3105.411465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3105.448399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.175933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.369066] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3116.528099] LustreError: 111209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.557590] LustreError: 111209:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3120.082212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.420980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3125.384090] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3133.939109] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3139.593713] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3145.774623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3146.067965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.944536] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3152.253059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3159.495882] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3167.522421] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3170.848904] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3174.882178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.892568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3174.904911] Lustre: Skipped 1 previous similar message [ 3174.911212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3174.916205] Lustre: Skipped 1 previous similar message [ 3180.020283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3180.032027] Lustre: Skipped 1 previous similar message [ 3180.482822] Lustre: server umount lustre-OST0000 complete [ 3185.125092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.133879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.142109] Lustre: Skipped 1 previous similar message [ 3190.263977] LustreError: 111205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.281246] LustreError: 111205:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3190.403209] Lustre: server umount lustre-MDT0000 complete [ 3194.101192] LustreError: 113739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948684 with bad export cookie 8304097905770237262 [ 3194.102842] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3194.110423] LustreError: 113739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3194.563710] Lustre: server umount lustre-MDT0001 complete [ 3207.537351] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 3211.214854] Key type lgssc unregistered [ 3211.643087] LNet: 114848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3211.651720] LNetError: 114848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3211.685809] LNet: Removed LNI 192.168.206.126@tcp [ 3212.861623] Key type .llcrypt unregistered [ 3212.866427] Key type ._llcrypt unregistered [ 3232.035153] Key type ._llcrypt registered [ 3232.039302] Key type .llcrypt registered [ 3232.155347] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 3233.136238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3233.191310] alg: No test for adler32 (adler32-zlib) [ 3234.228530] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3234.520486] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 3236.200201] Key type lgssc registered [ 3237.127703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3244.236389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3244.254582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.651356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3249.715246] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3255.783581] LustreError: 116085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.808914] LustreError: 116085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3257.796076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.220398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3261.858587] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3269.982544] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3276.063531] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3282.710500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.920943] LustreError: 117906:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.130325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.462466] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3290.610791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3297.187226] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3305.955776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.960616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.983747] Lustre: Skipped 1 previous similar message [ 3306.005681] Lustre: Skipped 1 previous similar message [ 3311.074593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3311.082563] Lustre: Skipped 1 previous similar message [ 3311.772729] Lustre: server umount lustre-OST0000 complete [ 3316.193278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.198814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.216189] Lustre: Skipped 1 previous similar message [ 3316.234223] Lustre: Skipped 1 previous similar message [ 3321.314146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3321.323442] Lustre: Skipped 1 previous similar message [ 3322.127644] Lustre: server umount lustre-MDT0000 complete [ 3325.847967] LustreError: 116067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948816 with bad export cookie 2367803777695531674 [ 3325.863253] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3326.356495] Lustre: server umount lustre-MDT0001 complete [ 3337.612221] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 3340.194653] Key type lgssc unregistered [ 3340.458367] LNet: 119615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3340.466044] LNetError: 119615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3340.483871] LNet: Removed LNI 192.168.206.126@tcp [ 3341.263162] Key type .llcrypt unregistered [ 3341.266495] Key type ._llcrypt unregistered [ 3363.083306] Key type ._llcrypt registered [ 3363.086531] Key type .llcrypt registered [ 3363.179808] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 3363.964348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3363.987752] alg: No test for adler32 (adler32-zlib) [ 3365.054210] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3365.273814] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 3367.016223] Key type lgssc registered [ 3368.033433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3374.913485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3374.943654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.379283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.650990] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3386.850769] LustreError: 121047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.878053] LustreError: 121047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3389.049587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3389.744899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3394.016746] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3400.106888] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3405.394634] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3412.786778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3412.988914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3420.502456] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3422.198440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 3428.535942] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3435.003768] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3439.165208] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3439.170475] Lustre: Skipped 2 previous similar messages [ 3443.305391] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3443.357434] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3444.024047] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3444.035304] Lustre: Skipped 2 previous similar messages [ 3451.969503] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3451.976735] Lustre: Skipped 1 previous similar message [ 3457.549503] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3457.558422] Lustre: Skipped 2 previous similar messages [ 3458.512596] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.26@tcp 192.168.0.[2,10]@tcp [ 3458.532355] Lustre: Skipped 1 previous similar message [ 3469.717981] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.26/24@tcp in log lustre-MDT0000 [ 3469.731568] Lustre: Skipped 2 previous similar messages [ 3475.526234] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.26/24@tcp [ 3481.570463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3481.583678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.591956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3483.619421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3483.621712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3483.640299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3483.649409] Lustre: Skipped 1 previous similar message [ 3487.225125] Lustre: server umount lustre-OST0000 complete [ 3492.323520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3492.337225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.369304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3496.947745] Lustre: server umount lustre-MDT0000 complete [ 3498.981154] LustreError: 121043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3499.001789] LustreError: 121043:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3500.730341] LustreError: 121781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948991 with bad export cookie 17543682467431822339 [ 3500.731566] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3500.740045] LustreError: 121781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3501.186304] Lustre: server umount lustre-MDT0001 complete [ 3513.009973] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 3515.572562] Key type lgssc unregistered [ 3515.877746] LNet: 125944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3515.898165] LNetError: 125944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3515.920980] LNet: Removed LNI 192.168.206.126@tcp [ 3516.986619] Key type .llcrypt unregistered [ 3516.990385] Key type ._llcrypt unregistered [ 3545.528964] Key type ._llcrypt registered [ 3545.531420] Key type .llcrypt registered [ 3545.652493] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 3546.761871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3546.803589] alg: No test for adler32 (adler32-zlib) [ 3547.923392] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3548.227372] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 3549.977296] Key type lgssc registered [ 3551.384310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3558.700796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3558.726294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.295112] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3560.322508] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.26/24@tcp [ 3560.330978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.198997] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3569.637952] LustreError: 127521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.669398] LustreError: 127521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3573.749463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.001765] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3574.036946] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.26/24@tcp [ 3574.088335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3578.332580] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3585.472497] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3590.936824] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3597.384659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3597.699973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.948039] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3604.918122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 3611.298468] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3624.418074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3624.426842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.443485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3625.443562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3625.448736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.450352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3627.770298] Lustre: server umount lustre-OST0000 complete [ 3635.682226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.688864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3635.697903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3635.704813] Lustre: Skipped 1 previous similar message [ 3637.627193] Lustre: server umount lustre-MDT0000 complete [ 3640.815101] LustreError: 127522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.838333] LustreError: 127522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3641.104236] LustreError: 128261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949131 with bad export cookie 1044362197843488634 [ 3641.115357] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3641.118904] LustreError: 128261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3641.498581] Lustre: server umount lustre-MDT0001 complete [ 3651.382430] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 3653.657696] Key type lgssc unregistered [ 3653.904960] LNet: 131105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3653.911283] LNetError: 131105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3653.926097] LNet: Removed LNI 192.168.206.126@tcp [ 3654.705886] Key type .llcrypt unregistered [ 3654.713190] Key type ._llcrypt unregistered [ 3680.765684] Key type ._llcrypt registered [ 3680.769095] Key type .llcrypt registered [ 3680.887143] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 3696.251627] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 3697.057519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3697.157581] alg: No test for adler32 (adler32-zlib) [ 3698.203227] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3698.438636] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 3700.096354] Key type lgssc registered [ 3701.163957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3740.090108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3740.162664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3741.549286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3741.622458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3741.764472] Lustre: lustre-MDT0000: new disk, initializing [ 3741.890861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.916715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3747.210399] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3758.243450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.345572] Lustre: 134524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3758.455351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3758.465475] Lustre: Skipped 1 previous similar message [ 3758.578867] Lustre: lustre-MDT0001: new disk, initializing [ 3758.707203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3758.766841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3758.781988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3763.273980] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3772.961237] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3779.001719] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3787.888752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3788.414560] Lustre: lustre-OST0000: new disk, initializing [ 3788.418712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3788.427893] Lustre: 136493:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3788.544195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.962256] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3798.122366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3798.157333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3798.435884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3808.893438] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3814.164493] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3814.332506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3818.969592] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3819.152228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3824.099327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3824.108646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.109239] LustreError: Skipped 1 previous similar message [ 3824.129550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3824.129821] Lustre: Skipped 1 previous similar message [ 3824.147445] Lustre: Skipped 1 previous similar message [ 3829.248045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.258721] Lustre: Skipped 1 previous similar message [ 3834.338605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3834.344349] Lustre: Skipped 1 previous similar message [ 3835.361438] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3835.613096] Lustre: server umount lustre-OST0000 complete [ 3839.458623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.470754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.474845] Lustre: Skipped 1 previous similar message [ 3839.492977] Lustre: Skipped 1 previous similar message [ 3844.577762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3844.583473] Lustre: Skipped 1 previous similar message [ 3845.812924] Lustre: server umount lustre-MDT0000 complete [ 3849.396509] LustreError: 134514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949340 with bad export cookie 9768293817515126457 [ 3849.403793] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.407103] LustreError: 134514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3849.732716] Lustre: server umount lustre-MDT0001 complete [ 3857.796432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.416911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.892983] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3863.524805] LustreError: 138712:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.541502] LustreError: 138712:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3868.653636] LustreError: 138713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.720261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3872.242047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3876.657144] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3883.812475] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3889.786394] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3896.942739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3897.349606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.544975] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3910.876062] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3915.519204] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3915.749105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3920.362657] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3920.583679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3930.613618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3930.781298] Lustre: lustre-OST0001: new disk, initializing [ 3930.785404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3930.791398] Lustre: 142083:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3930.937294] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3937.410120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3937.430368] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3937.539248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3938.155086] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3947.546726] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3947.711710] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3952.621588] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3952.875603] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3958.473268] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91c306a63000.ost_server_uuid 50 [ 3959.987791] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91c306a63000.ost_server_uuid in IDLE state after 0 sec [ 3968.490985] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3968.504645] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3968.510495] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3968.518607] Lustre: Skipped 1 previous similar message [ 3973.568175] Lustre: server umount lustre-OST0001 complete [ 3973.612275] LustreError: 142391:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.643382] LustreError: 142391:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3978.217287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3978.227429] LustreError: Skipped 1 previous similar message [ 3978.235976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.723125] LustreError: 140533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3978.771621] LustreError: 140533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3983.421769] Lustre: server umount lustre-OST0000 complete [ 3988.961579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.963121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3988.970815] Lustre: Skipped 2 previous similar messages [ 3988.993128] Lustre: Skipped 4 previous similar messages [ 3993.648716] Lustre: server umount lustre-MDT0000 complete [ 3994.083732] LustreError: 139927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.106290] LustreError: 139927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3997.687806] LustreError: 138693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949488 with bad export cookie 9768293817515127339 [ 3997.689650] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3997.695095] LustreError: 138693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3998.319598] Lustre: server umount lustre-MDT0001 complete [ 4017.671791] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 4020.635476] Key type lgssc unregistered [ 4021.034871] LNet: 145085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4021.049100] LNetError: 145085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4021.065777] LNet: Removed LNI 192.168.206.126@tcp [ 4022.015839] Key type .llcrypt unregistered [ 4022.018928] Key type ._llcrypt unregistered [ 4063.156853] Key type ._llcrypt registered [ 4063.160132] Key type .llcrypt registered [ 4063.267992] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 4077.843747] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 4079.260086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4079.371924] alg: No test for adler32 (adler32-zlib) [ 4080.477662] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 4080.708334] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 4082.376278] Key type lgssc registered [ 4083.608812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4119.125530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4119.225738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4120.477549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4120.507558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4120.576842] Lustre: lustre-MDT0000: new disk, initializing [ 4120.658368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.682718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4125.516882] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4136.448397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.680833] Lustre: 149255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4136.770259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4136.779317] Lustre: Skipped 1 previous similar message [ 4136.918812] Lustre: lustre-MDT0001: new disk, initializing [ 4137.073214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4137.133185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4137.145954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4141.889669] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4152.690379] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4158.610041] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4167.291542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.706154] Lustre: lustre-OST0000: new disk, initializing [ 4167.711395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4167.724449] Lustre: 151224:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4167.877604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.664884] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4177.948646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4177.953840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4178.044087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4185.373683] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4190.851521] Lustre: Failing over lustre-OST0000 [ 4191.037904] Lustre: server umount lustre-OST0000 complete [ 4208.608937] Lustre: 147469:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790949683/real 1790949683] req@00000000b21855b4 x1877946737396096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790949699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4208.613879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.646873] Lustre: 147469:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4208.684486] Lustre: Skipped 1 previous similar message [ 4209.633284] Lustre: 147468:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790949684/real 1790949684] req@0000000075e0e3ad x1877946737396480/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790949700 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4209.673421] Lustre: 147468:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4210.642678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4210.961080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.988370] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4215.008147] Lustre: 147469:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790949689/real 1790949689] req@00000000e70c92e7 x1877946737397504/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790949705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4215.044855] Lustre: 147469:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4217.474978] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4219.114863] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4220.003027] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4220.005425] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4221.093683] Lustre: 147468:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790949694/real 1790949694] req@0000000007836ebb x1877946737398656/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790949710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4221.323249] Lustre: Failing over lustre-MDT0000 [ 4221.710962] Lustre: server umount lustre-MDT0000 complete [ 4223.980296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4223.986167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.602629] LustreError: 149266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.621917] LustreError: 149266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4230.224178] LustreError: 152648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.724915] LustreError: 149261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.756127] LustreError: 149261:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4239.844394] LustreError: 149261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.873880] LustreError: 149261:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4240.612670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4240.746902] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4241.024118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4241.085993] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4241.584490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4246.038420] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4246.516303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4246.523826] Lustre: Skipped 1 previous similar message [ 4246.558686] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4246.605077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4256.225410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4256.244447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.256769] Lustre: Skipped 2 previous similar messages [ 4256.261593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4260.286693] Lustre: server umount lustre-OST0000 complete [ 4266.982656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4266.988956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.990879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4266.990886] Lustre: Skipped 1 previous similar message [ 4267.027953] Lustre: Skipped 2 previous similar messages [ 4272.097588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4272.110597] Lustre: Skipped 2 previous similar messages [ 4277.217964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4277.231104] Lustre: Skipped 1 previous similar message [ 4279.264147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4279.422747] Lustre: server umount lustre-MDT0000 complete [ 4282.342104] LustreError: 150463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.366727] LustreError: 150463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4284.076727] LustreError: 149248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949774 with bad export cookie 16877465510307366913 [ 4284.084473] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4284.096936] LustreError: 149248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4284.602090] Lustre: server umount lustre-MDT0001 complete [ 4295.968278] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 4299.242782] Key type lgssc unregistered [ 4299.563593] LNet: 154892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4299.571335] LNetError: 154892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4299.588373] LNet: Removed LNI 192.168.206.126@tcp [ 4300.404445] Key type .llcrypt unregistered [ 4300.411755] Key type ._llcrypt unregistered [ 4337.728238] Key type ._llcrypt registered [ 4337.731150] Key type .llcrypt registered [ 4337.851850] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 4339.035323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4339.080208] alg: No test for adler32 (adler32-zlib) [ 4340.096487] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 4340.308058] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 4341.960248] Key type lgssc registered [ 4343.045800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4351.487509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4351.503828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.967327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.247979] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4363.234753] LustreError: 156431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.261101] LustreError: 156431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4367.289138] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4368.354129] LustreError: 156430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.586247] LustreError: 156459:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4371.599601] LustreError: 156459:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4371.605598] LustreError: 156459:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4371.909880] Lustre: server umount lustre-MDT0000 complete [ 4389.361505] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 4400.892747] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 4441.021597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.577367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4441.669976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4441.835883] Lustre: lustre-MDT0000: new disk, initializing [ 4441.970441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.987534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4446.403087] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4457.968932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.107932] Lustre: 159764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4458.162954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4458.166906] Lustre: Skipped 1 previous similar message [ 4458.286594] Lustre: lustre-MDT0001: new disk, initializing [ 4458.429288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4458.471546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4458.494828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4464.089620] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4473.842164] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4479.662752] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4486.730405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4486.955826] Lustre: lustre-OST0000: new disk, initializing [ 4486.962164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4486.966890] Lustre: 161730:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4487.093838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.682119] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4495.463978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4495.481213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4495.598332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4503.441637] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4508.260711] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4510.182887] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:06:38 (1790949998) [ 4770.785949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4770.799629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4770.820183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.322484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.324185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.360117] Lustre: Skipped 1 previous similar message [ 4772.368120] Lustre: Skipped 2 previous similar messages [ 4776.284670] Lustre: server umount lustre-MDT0000 complete [ 4777.443687] LustreError: 159774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.476747] LustreError: 159774:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4781.306327] LustreError: 159755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950271 with bad export cookie 16017988283314229282 [ 4781.310400] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4781.330402] LustreError: 159755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4782.560914] LustreError: 159770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.563416] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.600940] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4786.599451] LustreError: 159771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.601754] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4787.906240] Lustre: server umount lustre-MDT0001 complete [ 4798.517665] Lustre: server umount lustre-OST0000 complete [ 4809.004883] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 4818.239694] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 4855.656424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.834797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4855.854826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4855.927974] Lustre: lustre-MDT0000: new disk, initializing [ 4856.016561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4856.033867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4860.833567] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4872.454486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.582501] Lustre: 166396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4872.649147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4872.652510] Lustre: Skipped 1 previous similar message [ 4872.705226] Lustre: lustre-MDT0001: new disk, initializing [ 4872.783583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4872.813548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4872.828711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4877.357676] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4887.241260] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4892.961327] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4899.765839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4900.061338] Lustre: lustre-OST0000: new disk, initializing [ 4900.065591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4900.073921] Lustre: 168363:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4900.155259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.361302] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4908.093944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4908.098974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4908.223510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4917.618759] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4922.654463] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4922.914337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4927.657076] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4927.901597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4934.116888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4934.119588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4934.141821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4944.352225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4944.360175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4944.375292] Lustre: Skipped 4 previous similar messages [ 4944.576981] Lustre: server umount lustre-OST0000 complete [ 4949.477676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4949.487134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4949.488345] Lustre: Skipped 2 previous similar messages [ 4963.296157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4963.505651] Lustre: server umount lustre-MDT0000 complete [ 4964.834657] LustreError: 166409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.852816] LustreError: 166409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4967.561606] LustreError: 166388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950458 with bad export cookie 16017988283314388112 [ 4967.575259] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.575839] LustreError: 166388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4967.927672] Lustre: server umount lustre-MDT0001 complete [ 4977.682586] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:14:26 (1790950466) [ 4995.314414] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 5006.700669] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 5046.099619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.356382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5046.403910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5046.482336] Lustre: lustre-MDT0000: new disk, initializing [ 5046.625573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5046.678337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5051.423979] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5062.137465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5062.261440] Lustre: 173122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5062.610799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5067.780288] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5078.326475] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5084.505655] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5091.141341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5091.486383] Lustre: lustre-OST0000: new disk, initializing [ 5091.490654] Lustre: Skipped 1 previous similar message [ 5091.496370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5091.499933] Lustre: Skipped 2 previous similar messages [ 5091.509671] Lustre: 175102:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5098.056610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5098.067217] Lustre: Skipped 1 previous similar message [ 5098.072974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5098.169213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5098.333569] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5107.761688] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5118.948835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5118.950992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.966572] LustreError: Skipped 1 previous similar message [ 5118.989732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5119.009777] Lustre: Skipped 7 previous similar messages [ 5120.711355] Lustre: server umount lustre-OST0000 complete [ 5130.919902] Lustre: server umount lustre-MDT0000 complete [ 5134.306634] LustreError: 174516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.321882] LustreError: 174516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5134.915859] LustreError: 173115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950625 with bad export cookie 16017988283314388994 [ 5134.924153] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5134.925750] LustreError: 173115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5135.315285] Lustre: server umount lustre-MDT0001 complete [ 5144.301149] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:17:12 (1790950632) [ 5160.873111] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 5170.658499] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 5207.500478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5207.694176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5207.715645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5207.776875] Lustre: lustre-MDT0000: new disk, initializing [ 5207.836119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5207.842252] Lustre: Skipped 2 previous similar messages [ 5207.854971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5213.123653] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5224.710046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.876091] Lustre: 179426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5225.107613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5229.278705] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5238.250613] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5243.699922] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5250.853353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5251.293671] Lustre: 181404:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5257.292391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5257.496176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5258.987871] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5270.768461] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5283.303178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5283.308820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.318192] LustreError: Skipped 1 previous similar message [ 5283.323258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5283.323266] Lustre: Skipped 2 previous similar messages [ 5283.369724] Lustre: Skipped 4 previous similar messages [ 5285.426622] Lustre: server umount lustre-OST0000 complete [ 5291.489773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5304.288565] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5304.507373] Lustre: server umount lustre-MDT0000 complete [ 5308.492300] LustreError: 179418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950799 with bad export cookie 16017988283314390079 [ 5308.500345] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.506899] LustreError: 179418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5308.960920] Lustre: server umount lustre-MDT0001 complete [ 5320.722095] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 5323.901532] Key type lgssc unregistered [ 5324.190188] LNet: 183260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5324.200381] LNetError: 183260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5324.226674] LNet: Removed LNI 192.168.206.126@tcp [ 5324.962959] Key type .llcrypt unregistered [ 5324.965428] Key type ._llcrypt unregistered [ 5351.926622] Key type ._llcrypt registered [ 5351.932957] Key type .llcrypt registered [ 5352.070216] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 5353.370706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5353.585878] alg: No test for adler32 (adler32-zlib) [ 5354.768371] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 5355.057714] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 5356.800223] Key type lgssc registered [ 5358.147719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5365.886376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5365.922260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.380920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.905794] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5377.514174] LustreError: 184692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.530911] LustreError: 184692:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5380.481774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.889735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5385.166980] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5391.983058] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5396.820893] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5403.444936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5403.952411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.472466] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5413.238564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5418.418507] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5437.921634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5437.939636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5437.970708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.955938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.966252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.979572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.094164] Lustre: server umount lustre-OST0000 complete [ 5447.649841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5447.668786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5447.700089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5447.705243] Lustre: Skipped 1 previous similar message [ 5453.288610] Lustre: server umount lustre-MDT0000 complete [ 5454.306450] LustreError: 184686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.326526] LustreError: 184686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5456.474182] LustreError: 184673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950947 with bad export cookie 8303781603937982009 [ 5456.479439] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5456.484881] LustreError: 184673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5456.806700] Lustre: server umount lustre-MDT0001 complete [ 5466.462191] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 5469.118373] Key type lgssc unregistered [ 5469.404511] LNet: 188246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5469.410748] LNetError: 188246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5469.425927] LNet: Removed LNI 192.168.206.126@tcp [ 5470.373856] Key type .llcrypt unregistered [ 5470.377214] Key type ._llcrypt unregistered [ 5491.330805] Key type ._llcrypt registered [ 5491.333327] Key type .llcrypt registered [ 5491.462046] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 5492.752958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5492.773923] alg: No test for adler32 (adler32-zlib) [ 5493.882933] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 5494.207368] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 5495.880446] Key type lgssc registered [ 5497.070060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5505.587849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5505.629919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.134377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.627518] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5517.283438] LustreError: 189658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.302129] LustreError: 189658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5519.409054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.698099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5523.790699] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5530.483280] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5535.223383] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5541.839465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5542.197122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.164842] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5551.433106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5556.087828] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5561.826078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5561.829902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.839277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5561.861539] Lustre: Skipped 1 previous similar message [ 5562.978120] Lustre: lustre-OST0000: Not available for connect from 192.168.206.26@tcp (stopping) [ 5562.984150] Lustre: Skipped 1 previous similar message [ 5566.949658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5567.414943] Lustre: server umount lustre-OST0000 complete [ 5574.656239] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff91c3124d5000.ost_server_uuid 50 [ 5576.054150] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5591.521708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5591.529830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.549264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5591.555401] Lustre: Skipped 1 previous similar message [ 5592.548094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.009371] Lustre: server umount lustre-MDT0000 complete [ 5598.690736] LustreError: 190706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5598.714282] LustreError: 190706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5598.870031] LustreError: 189638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951089 with bad export cookie 13167660463605158596 [ 5598.871400] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5598.882293] LustreError: 189638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5599.192563] Lustre: server umount lustre-MDT0001 complete [ 5607.692517] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:24:56 (1790951096) [ 5614.098533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.441996] LustreError: 193571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.505316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.700642] LustreError: 193572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5620.308791] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5624.802049] LustreError: 193571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.761060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.206743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5635.047352] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5643.346599] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5649.631897] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5657.279777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5657.764727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.820324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5664.024888] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5672.022479] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5679.619228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5679.903315] Lustre: lustre-OST0001: new disk, initializing [ 5679.910408] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5679.923257] Lustre: 196414:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5680.052280] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5686.508020] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5688.404175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5688.411480] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5688.531690] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5695.808640] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5699.050349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.074152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5699.088123] Lustre: Skipped 5 previous similar messages [ 5704.780812] Lustre: server umount lustre-OST0000 complete [ 5706.840588] LustreError: 195394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.876963] LustreError: 195394:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5712.123717] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5712.327500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5717.085889] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5717.279413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5724.646923] LustreError: 196879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.668326] LustreError: 196879:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 5734.881183] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5734.883321] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.902916] LustreError: Skipped 1 previous similar message [ 5734.904116] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5734.904121] Lustre: Skipped 3 previous similar messages [ 5734.946461] Lustre: Skipped 2 previous similar messages [ 5737.502074] Lustre: server umount lustre-OST0001 complete [ 5743.075593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5743.087686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5754.338176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.353134] Lustre: Skipped 9 previous similar messages [ 5756.384305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5756.555243] Lustre: server umount lustre-MDT0000 complete [ 5759.460460] LustreError: 193571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.482146] LustreError: 193571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5760.541378] LustreError: 196413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951251 with bad export cookie 13167660463605159667 [ 5760.548099] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5760.554922] LustreError: 196413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5760.912926] Lustre: server umount lustre-MDT0001 complete [ 5789.023895] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:27:57 (1790951277) [ 5796.917638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.223388] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5797.287456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5797.504618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.425060] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5812.181170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.274447] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5812.297852] Lustre: 199856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5817.084330] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5824.892266] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5831.347628] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5839.536791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5839.850582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5845.692100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5847.699556] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5856.609939] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5864.663537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5864.731893] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5865.072442] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5865.084357] Lustre: Skipped 2 previous similar messages [ 5872.873258] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5873.571349] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5882.039062] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5886.434745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5886.454454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.470205] Lustre: Skipped 1 previous similar message [ 5886.476462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5891.496360] Lustre: server umount lustre-OST0000 complete [ 5893.195373] LustreError: 201708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.972558] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5908.983098] LustreError: Skipped 1 previous similar message [ 5913.570714] Lustre: server umount lustre-OST0001 complete [ 5919.202375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.215451] Lustre: Skipped 4 previous similar messages [ 5924.168863] Lustre: server umount lustre-MDT0000 complete [ 5927.961487] LustreError: 199848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951418 with bad export cookie 13167660463605161074 [ 5927.964753] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5927.972164] LustreError: 199848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5928.516698] Lustre: server umount lustre-MDT0001 complete [ 5953.271026] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:30:41 (1790951441) [ 5969.315054] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 5978.817942] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 6015.916094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6016.150558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6016.190189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6016.279644] Lustre: lustre-MDT0000: new disk, initializing [ 6016.369633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6016.388439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6020.876815] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6032.540643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6032.699420] Lustre: 207950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6032.792509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6032.800342] Lustre: Skipped 1 previous similar message [ 6032.918327] Lustre: lustre-MDT0001: new disk, initializing [ 6033.078986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6033.091833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6038.398359] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6050.697220] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6057.037234] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6065.294498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6065.731220] Lustre: lustre-OST0000: new disk, initializing [ 6065.734601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6065.741362] Lustre: 209916:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6071.396875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6071.408608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6071.523949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6073.080906] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6084.257873] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6089.821425] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6090.032614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6096.164620] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6096.522100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6102.498230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6102.500889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6102.511952] Lustre: Skipped 1 previous similar message [ 6102.517431] Lustre: Skipped 8 previous similar messages [ 6102.525037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6102.536441] LustreError: Skipped 1 previous similar message [ 6113.252856] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6113.477289] Lustre: server umount lustre-OST0000 complete [ 6132.192243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6132.363363] Lustre: server umount lustre-MDT0000 complete [ 6133.222459] LustreError: 209060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.271565] LustreError: 209060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6136.987436] LustreError: 207941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951627 with bad export cookie 13167660463605162859 [ 6136.988827] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6136.996579] LustreError: 207941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6137.409219] Lustre: server umount lustre-MDT0001 complete [ 6147.490717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6152.671817] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6162.413043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.844993] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6176.358716] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6183.358186] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6191.668261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6200.501951] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6208.071413] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6213.753299] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6214.099445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6219.115311] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6219.293235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6222.845206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6227.693422] Lustre: server umount lustre-OST0000 complete [ 6236.251656] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6241.134197] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6270.016366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6276.423265] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6283.393727] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6287.722446] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.755742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.767412] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6287.804246] Lustre: Skipped 1 previous similar message [ 6289.351490] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6298.088710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.090411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6298.100548] Lustre: Skipped 4 previous similar messages [ 6298.133605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6298.157975] Lustre: Skipped 14 previous similar messages [ 6299.594620] Lustre: server umount lustre-OST0000 complete [ 6309.696402] Lustre: server umount lustre-MDT0000 complete [ 6313.553238] LustreError: 216254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951804 with bad export cookie 13167660463605163741 [ 6313.554447] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6313.583469] LustreError: 216254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6313.918862] Lustre: server umount lustre-MDT0001 complete [ 6322.614753] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:36:51 (1790951811) [ 6329.009623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.759434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6329.766357] Lustre: Skipped 6 previous similar messages [ 6334.625755] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6343.002730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.560677] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6355.714261] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6361.001402] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6367.526314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6373.710163] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6381.501455] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6386.143451] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6386.399044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6391.545137] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6391.882158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6398.991314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.228615] Lustre: lustre-OST0001: new disk, initializing [ 6399.232790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6399.238866] Lustre: 221484:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6405.768948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6405.777728] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6405.898389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6406.610971] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6417.203224] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6421.411142] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6421.618951] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6426.199483] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6426.458288] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6431.713789] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6431.727279] LustreError: Skipped 2 previous similar messages [ 6434.756227] Lustre: server umount lustre-OST0001 complete [ 6436.843202] LustreError: 221788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6436.866368] LustreError: 221788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6441.549634] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6441.807461] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6447.478317] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6447.759744] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6477.821789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6483.455354] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6483.483493] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6483.488426] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6485.261735] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6492.954352] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6494.339745] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6502.222632] Lustre: server umount lustre-OST0001 complete [ 6524.890168] Lustre: server umount lustre-OST0000 complete [ 6534.716588] Lustre: server umount lustre-MDT0000 complete [ 6537.987753] LustreError: 224853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952028 with bad export cookie 13167660463605164700 [ 6537.987753] LustreError: 218096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952028 with bad export cookie 13167660463605164700 [ 6537.988036] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.383046] Lustre: server umount lustre-MDT0001 complete [ 6569.725864] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:40:57 (1790952057) [ 6577.832138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6578.122821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6578.169181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6582.788366] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6593.114780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.258529] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6593.306913] Lustre: 226727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6597.933970] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6605.770648] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6611.753600] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6618.327923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6618.629097] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6626.371569] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6636.289532] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6647.586361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.651198] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6654.059332] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6660.964982] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6662.340090] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6666.179085] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6666.353968] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6670.223262] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6670.414982] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6675.596509] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91c306d4a800.ost_server_uuid 50 [ 6677.500930] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91c306d4a800.ost_server_uuid in IDLE state after 0 sec [ 6678.438792] Lustre: Permanently deactivating lustre-OST0001 [ 6678.455526] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6682.167595] Lustre: Permanently reactivating lustre-OST0001 [ 6686.177684] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6686.187870] Lustre: Skipped 11 previous similar messages [ 6686.192810] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6686.197327] Lustre: Skipped 14 previous similar messages [ 6692.235397] Lustre: server umount lustre-OST0001 complete [ 6696.933993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6696.944505] LustreError: Skipped 5 previous similar messages [ 6701.700929] Lustre: server umount lustre-OST0000 complete [ 6711.845361] Lustre: server umount lustre-MDT0000 complete [ 6716.040190] LustreError: 226720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952206 with bad export cookie 13167660463605165995 [ 6716.041903] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6716.059520] LustreError: 226720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6716.529742] Lustre: server umount lustre-MDT0001 complete [ 6746.081384] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:43:54 (1790952234) [ 6754.977425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.293851] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6755.354502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6755.364696] Lustre: Skipped 5 previous similar messages [ 6760.875985] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6771.067349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.275768] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6771.339800] Lustre: 233427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6777.151981] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6785.351096] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6791.880954] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6799.181475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6799.487665] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6799.524184] Lustre: Permanently deactivating lustre-OST0000 [ 6799.538935] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6805.900439] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6809.120293] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6813.854662] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6819.830578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6819.918487] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6826.054841] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6828.286364] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6832.784931] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6834.614824] Lustre: Permanently reactivating lustre-OST0000 [ 6834.627350] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6834.631609] Lustre: Skipped 2 previous similar messages [ 6834.739067] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6834.752188] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6834.763279] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6834.771769] Lustre: Skipped 1 previous similar message [ 6834.828668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6834.853784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6850.622497] Lustre: server umount lustre-OST0001 complete [ 6860.912449] Lustre: server umount lustre-OST0000 complete [ 6872.744851] Lustre: server umount lustre-MDT0000 complete [ 6877.557797] LustreError: 233422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952368 with bad export cookie 13167660463605167647 [ 6877.579104] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6877.582967] LustreError: 233422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6878.063860] Lustre: server umount lustre-MDT0001 complete [ 6893.834585] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 6898.211963] Key type lgssc unregistered [ 6898.683190] LNet: 238310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6898.691158] LNetError: 238310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6898.713553] LNet: Removed LNI 192.168.206.126@tcp [ 6899.776384] Key type .llcrypt unregistered [ 6899.782269] Key type ._llcrypt unregistered [ 6921.354595] Key type ._llcrypt registered [ 6921.357095] Key type .llcrypt registered [ 6921.469801] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 6922.590338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6922.714572] alg: No test for adler32 (adler32-zlib) [ 6923.750228] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 6924.027826] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 6925.721087] Key type lgssc registered [ 6926.851893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6936.697775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6936.725773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6938.246479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.879795] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6947.812635] LustreError: 239783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6947.837578] LustreError: 239783:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6949.896962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.026351] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6950.050901] Lustre: 239773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6950.061731] Lustre: Permanently deactivating lustre-MDT0001 [ 6950.072568] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6950.364909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6953.492100] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6953.501462] LustreError: 239814:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6953.509803] LustreError: 239814:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6953.519697] LustreError: 239814:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6953.815504] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6961.450691] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6966.762806] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6972.727096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6973.033812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6975.078464] LustreError: 241626:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6975.098502] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6978.676299] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6980.578864] LustreError: 241626:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.636134] LustreError: 241763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.648851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6982.660415] LustreError: 241763:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6985.395670] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6987.747953] LustreError: 241763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.757784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6991.962464] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6996.981926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6996.993369] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6997.695676] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7007.767798] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7012.428600] Lustre: Permanently reactivating lustre-MDT0001 [ 7012.472804] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7012.487718] Lustre: Skipped 2 previous similar messages [ 7015.498460] Lustre: lustre-MDT0001: Client c6f14c98-f481-4601-bb03-03f17c2d3b96 (at 192.168.206.26@tcp) reconnecting [ 7023.350641] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7025.209076] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7032.588639] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7032.903716] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7038.936459] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7039.183714] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7042.777568] Lustre: Permanently deactivating lustre-MDT0001 [ 7042.798721] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7042.811502] Lustre: Skipped 2 previous similar messages [ 7049.294234] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7057.890543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7057.896817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.911662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7058.413811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.423155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7058.431861] Lustre: Skipped 2 previous similar messages [ 7058.438765] Lustre: Skipped 2 previous similar messages [ 7063.139190] Lustre: server umount lustre-MDT0000 complete [ 7063.521735] LustreError: 240529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7063.545353] LustreError: 240529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7066.393836] LustreError: 239764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952557 with bad export cookie 17051388363303610 [ 7066.395735] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7066.404069] LustreError: 239764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7066.960343] Lustre: server umount lustre-MDT0001 complete [ 7072.732743] Lustre: server umount lustre-OST0000 complete [ 7083.656387] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:49:32 (1790952572) [ 7093.780050] Lustre: server umount lustre-OST0001 complete [ 7101.076095] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 7111.159417] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7146.915238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7147.114424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7147.124566] Lustre: Skipped 2 previous similar messages [ 7147.142724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7147.205867] Lustre: lustre-MDT0000: new disk, initializing [ 7147.265982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7147.280332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7151.389743] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7161.825889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.964678] Lustre: 248043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7162.029086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7162.036118] Lustre: Skipped 1 previous similar message [ 7162.132367] Lustre: lustre-MDT0001: new disk, initializing [ 7162.231198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7162.285816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7162.295150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7166.735600] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7176.802860] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7182.621094] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7189.687304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7189.970621] Lustre: lustre-OST0000: new disk, initializing [ 7189.975516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7189.982838] Lustre: 250011:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7190.099367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.033357] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7196.733815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7196.753337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7196.862661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7205.690893] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7208.112906] LustreError: 248048:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7218.146444] LustreError: 248048:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7218.161981] LustreError: 249951:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7219.521670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7219.784090] Lustre: lustre-OST0001: new disk, initializing [ 7219.789864] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7219.799762] Lustre: 251126:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7228.241122] LustreError: 249951:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7228.289312] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7228.987858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7228.996725] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7229.144039] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7238.714786] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7240.579524] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7244.784120] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.800368] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7244.810845] Lustre: Skipped 1 previous similar message [ 7244.811458] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7244.811477] Lustre: Skipped 1 previous similar message [ 7244.849838] LustreError: Skipped 1 previous similar message [ 7249.229651] Lustre: server umount lustre-OST0001 complete [ 7249.895912] LustreError: 251546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.913768] LustreError: 251546:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7268.835516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7268.835808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.846174] LustreError: Skipped 1 previous similar message [ 7268.872703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7268.880320] Lustre: Skipped 2 previous similar messages [ 7272.186609] Lustre: server umount lustre-OST0000 complete [ 7280.100112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7280.117328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.141314] Lustre: Skipped 1 previous similar message [ 7280.145623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7280.147655] Lustre: Skipped 2 previous similar messages [ 7281.928678] Lustre: server umount lustre-MDT0000 complete [ 7285.738905] LustreError: 248054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7285.779615] LustreError: 248054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 7285.965739] LustreError: 248033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952776 with bad export cookie 17051388363306424 [ 7285.969894] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7285.974413] LustreError: 248033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7286.016297] LustreError: 238986:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 7292.417463] Lustre: server umount lustre-MDT0001 complete [ 7302.582209] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 7305.183493] Key type lgssc unregistered [ 7305.451584] LNet: 253159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7305.473753] LNetError: 253159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7305.489395] LNet: Removed LNI 192.168.206.126@tcp [ 7306.417147] Key type .llcrypt unregistered [ 7306.419800] Key type ._llcrypt unregistered [ 7344.897922] Key type ._llcrypt registered [ 7344.901449] Key type .llcrypt registered [ 7345.034814] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7346.194626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7346.330312] alg: No test for adler32 (adler32-zlib) [ 7347.403560] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 7347.623926] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 7349.288248] Key type lgssc registered [ 7350.587335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7359.038412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7359.051920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7360.352665] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7360.387654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7360.684099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.321853] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7375.860642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7375.945099] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7375.979505] Lustre: 255383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7376.288501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7380.361645] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7387.533618] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7392.701543] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7398.398799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7398.600410] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7398.891704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.439322] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7407.740095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7411.331718] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7458.786583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7458.795561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7458.824870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7459.306349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7459.307031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.331929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7459.334941] Lustre: Skipped 1 previous similar message [ 7463.004878] Lustre: server umount lustre-OST0000 complete [ 7467.319589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7478.841209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.221213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.238422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7480.256943] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7484.410588] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7484.441283] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7486.222706] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7499.345984] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7541.729467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7541.753060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.791961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.829040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.830943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.838801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.928425] Lustre: server umount lustre-OST0000 complete [ 7555.554331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7555.567520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.589857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7555.599892] Lustre: Skipped 1 previous similar message [ 7566.304651] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7566.309143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7566.333554] Lustre: Skipped 5 previous similar messages [ 7566.483722] Lustre: server umount lustre-MDT0000 complete [ 7569.717541] LustreError: 255376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953060 with bad export cookie 9356518764366682124 [ 7569.724386] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7569.728442] LustreError: 255376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7570.069989] Lustre: server umount lustre-MDT0001 complete [ 7581.714831] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 7584.412069] Key type lgssc unregistered [ 7584.704370] LNet: 260809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7584.715441] LNetError: 260809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7584.733645] LNet: Removed LNI 192.168.206.126@tcp [ 7585.565442] Key type .llcrypt unregistered [ 7585.568667] Key type ._llcrypt unregistered [ 7611.753919] Key type ._llcrypt registered [ 7611.757722] Key type .llcrypt registered [ 7611.902804] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7613.732969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7613.793409] alg: No test for adler32 (adler32-zlib) [ 7614.963779] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 7615.291168] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 7617.008184] Key type lgssc registered [ 7618.101405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7625.237326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7625.262877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.715532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.115944] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7636.962911] LustreError: 262240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7637.005581] LustreError: 262240:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7638.010873] LustreError: 262241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7643.107851] LustreError: 262240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.045912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7644.277103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.451859] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7661.896154] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7667.668899] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7675.116660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7675.407527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.359302] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7688.869788] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7697.988802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7723.496637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7723.506725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7723.515170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7723.559082] Lustre: Skipped 1 previous similar message [ 7726.133851] Lustre: server umount lustre-OST0000 complete [ 7731.169995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7731.181479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.190952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.194557] Lustre: Skipped 1 previous similar message [ 7733.733525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7733.734621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.364752] Lustre: server umount lustre-MDT0000 complete [ 7738.849725] LustreError: 262240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.895317] LustreError: 262240:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7741.522296] LustreError: 262222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953232 with bad export cookie 9032928428081537275 [ 7741.533033] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7741.534299] LustreError: 262222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7742.120370] Lustre: server umount lustre-MDT0001 complete [ 7753.935042] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 7756.785482] Key type lgssc unregistered [ 7756.989821] LNet: 266567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7756.998861] LNetError: 266567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7758.069207] LNet: Removed LNI 192.168.206.126@tcp [ 7758.920619] Key type .llcrypt unregistered [ 7758.922548] Key type ._llcrypt unregistered [ 7777.217190] Key type ._llcrypt registered [ 7777.219412] Key type .llcrypt registered [ 7777.307398] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7778.739859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7778.778446] alg: No test for adler32 (adler32-zlib) [ 7779.882555] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 7780.055710] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 7781.720901] Key type lgssc registered [ 7782.871446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7790.009505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7790.054567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.582500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.536277] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7801.828388] LustreError: 267817:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.857615] LustreError: 267817:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7805.697052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7805.999771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7810.233728] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7816.736759] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7822.140914] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7828.852835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.259273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7834.090777] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7836.475925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7840.435453] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7852.005134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7852.005208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.012411] LustreError: Skipped 1 previous similar message [ 7852.015198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7852.026059] Lustre: Skipped 1 previous similar message [ 7855.790145] Lustre: server umount lustre-OST0000 complete [ 7862.241574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7862.246574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7862.247390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7862.258794] Lustre: Skipped 1 previous similar message [ 7862.274178] Lustre: Skipped 2 previous similar messages [ 7865.542340] Lustre: server umount lustre-MDT0000 complete [ 7867.364520] LustreError: 268860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7867.377486] LustreError: 268860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7868.676541] LustreError: 267799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953359 with bad export cookie 10486653184023020634 [ 7868.686457] LustreError: 267799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7868.686731] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7869.080840] Lustre: server umount lustre-MDT0001 complete [ 7879.017078] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 7882.172479] Key type lgssc unregistered [ 7882.542626] LNet: 271494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7882.553475] LNetError: 271494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7882.575386] LNet: Removed LNI 192.168.206.126@tcp [ 7883.451975] Key type .llcrypt unregistered [ 7883.453767] Key type ._llcrypt unregistered [ 7907.847329] Key type ._llcrypt registered [ 7907.849433] Key type .llcrypt registered [ 7907.942286] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7908.968952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7909.002551] alg: No test for adler32 (adler32-zlib) [ 7909.938242] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 7910.084883] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 7911.712538] Key type lgssc registered [ 7913.005440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7919.784102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7919.822363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.180732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7925.516353] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7930.338706] LustreError: 272925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7930.351698] LustreError: 272925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7932.678811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.945691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7937.029258] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7944.924131] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7950.430317] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7957.270148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7957.480957] LustreError: 274746:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7957.665629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7964.618227] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7966.256878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7973.810944] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7998.946579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7998.973754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7998.996583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8002.034207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8002.037489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.782390] Lustre: server umount lustre-OST0000 complete [ 8009.698824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8009.716632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8009.744233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.755344] Lustre: Skipped 1 previous similar message [ 8012.257646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8012.260510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.248912] Lustre: server umount lustre-MDT0000 complete [ 8017.378330] LustreError: 272926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.420884] LustreError: 272926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8019.634710] LustreError: 272905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953510 with bad export cookie 7186241158955153860 [ 8019.645328] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8019.651440] LustreError: 272905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8020.004675] Lustre: server umount lustre-MDT0001 complete [ 8030.484759] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 8033.491305] Key type lgssc unregistered [ 8033.817160] LNet: 277233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8033.830656] LNetError: 277233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8033.849347] LNet: Removed LNI 192.168.206.126@tcp [ 8034.958373] Key type .llcrypt unregistered [ 8034.963856] Key type ._llcrypt unregistered [ 8051.422629] Key type ._llcrypt registered [ 8051.424690] Key type .llcrypt registered [ 8051.533856] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8052.912222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8052.973230] alg: No test for adler32 (adler32-zlib) [ 8054.042610] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 8054.271732] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 8055.976236] Key type lgssc registered [ 8057.388710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8065.370589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8065.399311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.928807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.941444] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8077.295280] LustreError: 278484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.320589] LustreError: 278484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8079.844250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.353206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8085.576898] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8094.772777] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8101.431699] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8110.915804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8111.281220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.324878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8118.444962] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8127.211347] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8143.331667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8143.336168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8143.348788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8145.382098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8145.384235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8145.393464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8145.428096] Lustre: Skipped 1 previous similar message [ 8148.022420] Lustre: server umount lustre-OST0000 complete [ 8155.620843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.641171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.646648] Lustre: Skipped 1 previous similar message [ 8159.047812] Lustre: server umount lustre-MDT0000 complete [ 8160.737642] LustreError: 278482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8160.773707] LustreError: 278482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8163.404434] LustreError: 278450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953654 with bad export cookie 16206549625455936006 [ 8163.407408] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8163.424116] LustreError: 278450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8163.755327] Lustre: server umount lustre-MDT0001 complete [ 8173.397432] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 8175.988154] Key type lgssc unregistered [ 8176.255910] LNet: 282185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8176.270694] LNetError: 282185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8176.291240] LNet: Removed LNI 192.168.206.126@tcp [ 8177.090380] Key type .llcrypt unregistered [ 8177.093491] Key type ._llcrypt unregistered [ 8188.249671] Key type ._llcrypt registered [ 8188.253972] Key type .llcrypt registered [ 8188.386420] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8206.004469] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 8218.416168] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8219.422468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8219.450314] alg: No test for adler32 (adler32-zlib) [ 8220.496618] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 8220.719511] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 8222.394670] Key type lgssc registered [ 8223.371947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8252.623276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8252.651465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.998032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8254.038771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8254.135520] Lustre: lustre-MDT0000: new disk, initializing [ 8254.250910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.272501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8258.419141] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8269.000380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.139592] Lustre: 285730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8269.194590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8269.204133] Lustre: Skipped 1 previous similar message [ 8269.249758] Lustre: lustre-MDT0001: new disk, initializing [ 8269.341160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8269.393504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8269.403045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8273.685185] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8283.369421] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8288.322832] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8293.899569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8294.163315] Lustre: lustre-OST0000: new disk, initializing [ 8294.166814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8294.172590] Lustre: 287696:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8294.254172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.277991] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8301.121702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8301.136602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8301.251710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8308.257940] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8312.787376] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8312.943553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8317.568428] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8317.793928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8322.019345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8322.020581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8322.038447] Lustre: Skipped 1 previous similar message [ 8322.051459] Lustre: Skipped 1 previous similar message [ 8327.151537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8327.157728] Lustre: Skipped 1 previous similar message [ 8332.261872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8332.265328] Lustre: Skipped 1 previous similar message [ 8334.305896] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8334.548090] Lustre: server umount lustre-OST0000 complete [ 8340.962920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8340.987074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8341.012214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8342.510968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.930301] Lustre: server umount lustre-MDT0000 complete [ 8348.832201] LustreError: 285723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953839 with bad export cookie 591341179468096416 [ 8348.833496] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8348.848147] LustreError: 285723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8349.193151] Lustre: server umount lustre-MDT0001 complete [ 8356.251292] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:10:45 (1790953845) [ 8362.123720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8362.743184] LustreError: 290108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8362.758740] LustreError: 290108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8362.837527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8366.739809] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8368.107675] LustreError: 290109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.222697] LustreError: 290108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.499852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.739664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8378.678580] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8386.190981] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8391.161690] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8396.938199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8397.301735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8402.441589] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8408.432936] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8417.762483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8417.768156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8417.772140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8417.786261] Lustre: Skipped 1 previous similar message [ 8417.789263] Lustre: Skipped 2 previous similar messages [ 8423.288300] Lustre: server umount lustre-OST0000 complete [ 8428.004423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.025699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8428.033256] Lustre: Skipped 4 previous similar messages [ 8432.596346] Lustre: server umount lustre-MDT0000 complete [ 8433.124721] LustreError: 290109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8433.141620] LustreError: 290109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8435.984601] LustreError: 290090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953926 with bad export cookie 591341179468097298 [ 8435.993687] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8435.994474] LustreError: 290090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8436.285423] Lustre: server umount lustre-MDT0001 complete [ 8446.193809] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 8448.587978] Key type lgssc unregistered [ 8448.812854] LNet: 293641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8448.818928] LNetError: 293641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8448.830954] LNet: Removed LNI 192.168.206.126@tcp [ 8449.466092] Key type .llcrypt unregistered [ 8449.467712] Key type ._llcrypt unregistered [ 8477.260846] Key type ._llcrypt registered [ 8477.263547] Key type .llcrypt registered [ 8477.353337] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 8491.949837] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8492.899675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8492.926481] alg: No test for adler32 (adler32-zlib) [ 8493.990859] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 8494.227545] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 8495.904237] Key type lgssc registered [ 8497.064532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8539.891073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8539.947319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.149365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8541.199251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8541.283393] Lustre: lustre-MDT0000: new disk, initializing [ 8541.345790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8541.361189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8545.290971] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8556.259635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.380656] Lustre: 297480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8556.445655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8556.453829] Lustre: Skipped 1 previous similar message [ 8556.555885] Lustre: lustre-MDT0001: new disk, initializing [ 8556.685342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8556.732313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8556.756657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8560.839493] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8570.220321] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8575.676335] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8582.858365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8583.153492] Lustre: lustre-OST2710: new disk, initializing [ 8583.164301] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8583.171057] Lustre: 299450:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8583.245655] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8589.217726] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8591.417177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8591.426597] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8591.525146] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8598.553694] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8604.910757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8605.075310] Lustre: lustre-OST03e8: new disk, initializing [ 8605.078961] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8605.085145] Lustre: 300516:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8605.150295] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8610.924048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8610.944702] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8611.047113] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8611.214605] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8618.509520] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8624.834910] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8625.039530] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8629.363407] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8629.541910] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8633.746550] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8633.997876] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8638.338913] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8638.536150] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8644.065070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8644.077176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8644.091569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8644.803629] Lustre: server umount lustre-MDT0000 complete [ 8647.147988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.576586] LustreError: 297473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954139 with bad export cookie 15505191062263821640 [ 8648.587329] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.593909] LustreError: 297473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8648.947518] Lustre: server umount lustre-MDT0001 complete [ 8662.940035] Lustre: server umount lustre-OST2710 complete [ 8675.716933] Lustre: server umount lustre-OST03e8 complete [ 8694.607565] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 8703.331595] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8736.319467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.517165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8736.542735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8736.607609] Lustre: lustre-MDT0000: new disk, initializing [ 8736.674308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.692865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8740.677734] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8751.153229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.249754] Lustre: 305766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8751.300551] LustreError: 305777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8751.309449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8751.311169] LustreError: 305777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8751.313164] Lustre: Skipped 1 previous similar message [ 8751.370832] Lustre: lustre-MDT0001: new disk, initializing [ 8751.474366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8751.523694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8751.531428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8756.022476] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8764.483220] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8769.699619] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8775.974741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8776.123815] Lustre: lustre-OST0000: new disk, initializing [ 8776.126345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8776.131277] Lustre: 307734:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8776.202373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.784124] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8781.877348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8781.884789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8782.024029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8793.370542] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8798.716424] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8799.056978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8804.086908] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8804.275365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8807.911150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.913295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8807.917817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8807.919991] Lustre: Skipped 3 previous similar messages [ 8807.970416] LustreError: Skipped 1 previous similar message [ 8813.028361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8813.035885] Lustre: Skipped 2 previous similar messages [ 8818.159138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8818.176508] Lustre: Skipped 1 previous similar message [ 8820.192342] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8820.354587] Lustre: server umount lustre-OST0000 complete [ 8828.387929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.407523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8828.422071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8828.444172] Lustre: Skipped 1 previous similar message [ 8830.280431] Lustre: server umount lustre-MDT0000 complete [ 8833.508400] LustreError: 305778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8833.525518] LustreError: 305778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8833.928258] LustreError: 305758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954324 with bad export cookie 15505191062263823327 [ 8833.934305] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8833.939467] LustreError: 305758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8834.366193] Lustre: server umount lustre-MDT0001 complete [ 8841.450046] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:18:50 (1790954330) [ 8843.160843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8844.760362] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:18:53 (1790954333) [ 8847.404890] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8871.500896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8871.745788] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8871.771105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8871.938499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8876.199753] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8883.491716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8883.665294] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8883.706659] Lustre: 311591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8888.686216] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8910.390527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8910.554486] Lustre: Denying initial registration attempt from nid 192.168.206.126@tcp, specified as failover [ 8910.560319] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8910.566701] LustreError: 313043:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 8910.574916] LustreError: 313043:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8910.587992] LustreError: 313043:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8910.665486] Lustre: server umount lustre-OST0000 complete [ 8910.671338] LustreError: 313043:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8918.498981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8918.501032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8918.519971] Lustre: Skipped 2 previous similar messages [ 8931.296612] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8931.531305] Lustre: server umount lustre-MDT0000 complete [ 8933.864385] LustreError: 311600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8933.889338] LustreError: 311600:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8934.945906] LustreError: 311581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954425 with bad export cookie 15505191062263824209 [ 8934.957446] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8935.168243] Lustre: server umount lustre-MDT0001 complete [ 8944.487849] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 8947.087916] Key type lgssc unregistered [ 8947.475522] LNet: 314164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8947.491245] LNetError: 314164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8947.513249] LNet: Removed LNI 192.168.206.126@tcp [ 8948.694660] Key type .llcrypt unregistered [ 8948.697789] Key type ._llcrypt unregistered [ 8961.443101] Key type ._llcrypt registered [ 8961.445256] Key type .llcrypt registered [ 8961.543481] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8962.511867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8962.635049] alg: No test for adler32 (adler32-zlib) [ 8963.800861] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 8964.063761] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 8965.729455] Key type lgssc registered [ 8966.981673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9015.257790] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9024.784702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9024.801244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9026.114388] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9026.149226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9026.391231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9031.176577] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9039.820572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9039.957299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9039.996256] Lustre: 316941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9040.310064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9044.488338] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9051.535631] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9056.361108] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9075.564100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9075.713905] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9075.862418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.977878] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9086.177163] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9090.021158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9090.022388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9090.043658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9090.061522] Lustre: Skipped 1 previous similar message [ 9093.964558] Lustre: server umount lustre-OST0000 complete [ 9100.262201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9100.264317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9100.277900] Lustre: Skipped 2 previous similar messages [ 9103.417610] Lustre: server umount lustre-MDT0000 complete [ 9107.408128] LustreError: 317694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954598 with bad export cookie 5486242408976717582 [ 9107.425828] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9107.441671] LustreError: 317694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9108.192629] Lustre: server umount lustre-MDT0001 complete [ 9119.285124] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 9122.477345] Key type lgssc unregistered [ 9122.826868] LNet: 320516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9122.838750] LNetError: 320516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9122.857453] LNet: Removed LNI 192.168.206.126@tcp [ 9123.693966] Key type .llcrypt unregistered [ 9123.696709] Key type ._llcrypt unregistered [ 9142.685825] Key type ._llcrypt registered [ 9142.690139] Key type .llcrypt registered [ 9142.836680] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9143.673325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9143.802518] alg: No test for adler32 (adler32-zlib) [ 9145.128425] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 9145.502068] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 9147.296174] Key type lgssc registered [ 9149.204424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9158.885798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9158.917548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9160.277112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9165.349862] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9169.385706] LustreError: 321926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.401257] LustreError: 321926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9174.504271] LustreError: 321927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9175.522862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9176.069389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9180.226351] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9186.773586] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9191.491489] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9199.664599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9200.040947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9205.355336] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9212.165218] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9220.585340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9220.598300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.608964] Lustre: Skipped 2 previous similar messages [ 9220.625311] Lustre: Skipped 2 previous similar messages [ 9224.820603] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.26@tcp (stopping) [ 9226.676725] Lustre: server umount lustre-MDT0000 complete [ 9229.896672] LustreError: 321923:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9231.212529] LustreError: 321908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954721 with bad export cookie 4021457064246415898 [ 9231.217186] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9231.226499] LustreError: 321908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9235.015527] LustreError: 323769:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.068070] LustreError: 323769:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9235.086917] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.26@tcp (stopping) [ 9235.095116] Lustre: Skipped 3 previous similar messages [ 9235.942030] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9237.584412] Lustre: server umount lustre-MDT0001 complete [ 9244.613756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9252.710654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.395837] LustreError: 321226:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000035e0aba8 x1877952048440832/t0(0) o250->MGC192.168.206.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9255.630850] LustreError: 325518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9255.653507] LustreError: 325518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9255.883703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9255.941951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9261.284522] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9265.242869] LustreError: 325521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9265.280205] LustreError: 325521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9267.185762] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9272.256577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9272.873258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9277.945144] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9279.572995] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9288.498046] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9289.873541] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9295.512598] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9296.633300] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9301.994793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9302.009855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9302.043048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9302.048735] Lustre: Skipped 2 previous similar messages [ 9303.520898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9306.499046] Lustre: server umount lustre-OST0000 complete [ 9313.763123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9313.764063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9313.802029] Lustre: Skipped 1 previous similar message [ 9313.812876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9313.839298] Lustre: Skipped 3 previous similar messages [ 9316.114315] Lustre: server umount lustre-MDT0000 complete [ 9318.882968] LustreError: 325518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9318.892602] LustreError: 325518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 9319.287174] LustreError: 321907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954809 with bad export cookie 4021457064246423570 [ 9319.293444] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9319.294595] LustreError: 321907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9319.639470] Lustre: server umount lustre-MDT0001 complete [ 9330.613370] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 9333.099992] Key type lgssc unregistered [ 9333.374668] LNet: 328193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9333.380720] LNetError: 328193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9334.440898] LNet: Removed LNI 192.168.206.126@tcp [ 9335.165677] Key type .llcrypt unregistered [ 9335.172236] Key type ._llcrypt unregistered [ 9355.267961] Key type ._llcrypt registered [ 9355.271117] Key type .llcrypt registered [ 9355.362191] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9356.441279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9356.461384] alg: No test for adler32 (adler32-zlib) [ 9357.560632] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 9357.865409] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 9359.608209] Key type lgssc registered [ 9361.208529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9370.501514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9370.527219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9372.145641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9377.042210] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9381.354871] LustreError: 329607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9381.376547] LustreError: 329607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9385.251655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9385.562070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9389.576775] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9397.099913] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9404.024755] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9410.742365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9410.744361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9410.775394] Lustre: Skipped 1 previous similar message [ 9410.812826] Lustre: Skipped 1 previous similar message [ 9414.040722] Lustre: server umount lustre-MDT0000 complete [ 9416.161739] LustreError: 329607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9416.182046] LustreError: 329607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9417.888432] LustreError: 329587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954908 with bad export cookie 9906683425660864906 [ 9417.905737] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9417.909849] LustreError: 329587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9418.314129] Lustre: server umount lustre-MDT0001 complete [ 9436.421924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.886617] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9436.952458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9437.165228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9441.556210] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9450.046887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9450.184969] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9450.226976] Lustre: 332196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9450.620945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9455.482463] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9464.702986] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9473.101918] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9483.562328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9483.776037] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9483.783168] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9483.790565] LustreError: 334078:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9483.800528] LustreError: 334078:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9483.805802] LustreError: 334078:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9483.855527] Lustre: server umount lustre-OST0000 complete [ 9483.858219] LustreError: 334078:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9494.033735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9494.457771] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9494.617350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9500.244819] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9502.887462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9507.182679] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9514.707744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9514.803267] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9514.918223] Lustre: lustre-OST0001: new disk, initializing [ 9514.924878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9514.935884] Lustre: 335621:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9515.006501] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9520.770322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9520.776565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9520.932575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9521.260526] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9528.354748] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9531.362359] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9531.367371] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.370755] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9531.372376] Lustre: Skipped 1 previous similar message [ 9531.391534] Lustre: Skipped 1 previous similar message [ 9536.494351] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9536.504090] Lustre: Skipped 1 previous similar message [ 9537.284544] Lustre: server umount lustre-OST0001 complete [ 9541.602207] LustreError: 335033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9541.608830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9541.614510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9541.616233] LustreError: 335033:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9541.646042] Lustre: Skipped 1 previous similar message [ 9546.724653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9546.725619] LustreError: 334545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9546.733604] Lustre: Skipped 2 previous similar messages [ 9546.760316] LustreError: 334545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9551.851643] LustreError: 334546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.866113] LustreError: 334546:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9554.400166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9554.548905] Lustre: server umount lustre-OST0000 complete [ 9562.082048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9562.086125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9562.102747] Lustre: Skipped 1 previous similar message [ 9562.127349] Lustre: Skipped 3 previous similar messages [ 9573.349014] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9573.529809] Lustre: server umount lustre-MDT0000 complete [ 9577.174842] LustreError: 332187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955067 with bad export cookie 9906683425660865459 [ 9577.185863] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9577.199746] LustreError: 332187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9577.722167] Lustre: server umount lustre-MDT0001 complete [ 9589.222870] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 9592.336764] Key type lgssc unregistered [ 9592.621568] LNet: 337581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9592.626161] LNetError: 337581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9592.641523] LNet: Removed LNI 192.168.206.126@tcp [ 9593.349645] Key type .llcrypt unregistered [ 9593.352503] Key type ._llcrypt unregistered [ 9660.665772] Key type ._llcrypt registered [ 9660.667715] Key type .llcrypt registered [ 9660.776504] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 9673.109158] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9674.166023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9674.192545] alg: No test for adler32 (adler32-zlib) [ 9675.261173] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 9675.433897] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 9677.121890] Key type lgssc registered [ 9678.396409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9706.189374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9706.241428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.465347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9707.488735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9707.581903] Lustre: lustre-MDT0000: new disk, initializing [ 9707.658849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9707.674193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9711.686364] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9722.679531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.876325] Lustre: 342510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9722.954976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9722.961105] Lustre: Skipped 1 previous similar message [ 9723.070339] Lustre: lustre-MDT0001: new disk, initializing [ 9723.186578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9723.244451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9723.259959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9728.293245] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9738.280973] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9743.635256] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9750.174824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9750.481132] Lustre: lustre-OST0000: new disk, initializing [ 9750.490188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9750.497956] Lustre: 344478:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9750.639819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9757.257250] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9757.776929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9757.788086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9757.914762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9767.362614] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9772.629085] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9772.970543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9779.597647] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9779.984118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9783.782033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.782630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9783.787842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9783.822243] Lustre: Skipped 1 previous similar message [ 9783.892377] LustreError: Skipped 1 previous similar message [ 9788.946910] Lustre: server umount lustre-OST0000 complete [ 9794.022248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9794.031815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9794.039508] Lustre: Skipped 1 previous similar message [ 9794.050815] Lustre: Skipped 2 previous similar messages [ 9799.145872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9799.158433] Lustre: Skipped 1 previous similar message [ 9804.257825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9804.266358] Lustre: Skipped 1 previous similar message [ 9807.328179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9807.695944] Lustre: server umount lustre-MDT0000 complete [ 9811.832588] LustreError: 342500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955302 with bad export cookie 903911068293221989 [ 9811.840778] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9812.333250] Lustre: server umount lustre-MDT0001 complete [ 9822.552175] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:35:10 (1790955310) [ 9831.141673] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:35:19 (1790955319) [ 9838.101924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9838.516218] LustreError: 347236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9838.535859] LustreError: 347236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9838.604936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9842.077585] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9843.686438] LustreError: 347237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9847.804723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9848.056723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9851.380225] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9856.611264] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9861.220200] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9865.990645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9866.204502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9870.819963] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9876.149445] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9893.090521] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9894.757145] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9896.041840] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9897.495130] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9899.026833] Lustre: Failing over lustre-MDT0000 [ 9899.264837] Lustre: server umount lustre-MDT0000 complete [ 9899.490644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9899.496092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9899.506638] LustreError: 347237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.196447] LustreError: 349450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9904.098456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.111721] Lustre: Skipped 1 previous similar message [ 9908.296639] LustreError: 348042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9908.313849] LustreError: 348042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9915.393316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9915.467972] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9915.639842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9915.674893] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9918.540628] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9919.131301] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9921.003657] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9921.019158] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9921.045807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9926.454183] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9927.491508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9930.743953] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9932.297830] Lustre: Failing over lustre-MDT0000 [ 9932.491573] Lustre: server umount lustre-MDT0000 complete [ 9933.933814] LustreError: 347233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9933.951460] LustreError: 347233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 9936.357098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9936.361241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9936.381612] Lustre: Skipped 1 previous similar message [ 9949.568973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9949.648780] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9949.785116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9949.826926] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9953.000084] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9954.409821] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9954.793702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9954.805623] Lustre: Skipped 2 previous similar messages [ 9954.831935] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9954.867778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9959.558935] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9960.582379] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9965.028249] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9965.032078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9965.032840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9965.039479] Lustre: Skipped 1 previous similar message [ 9970.991534] Lustre: server umount lustre-MDT0000 complete [ 9974.220273] LustreError: 350028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955464 with bad export cookie 903911068293224698 [ 9974.228898] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9974.233359] LustreError: 350028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9974.513691] Lustre: server umount lustre-MDT0001 complete [ 9987.504358] Lustre: server umount lustre-OST0000 complete [ 9999.108800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9999.395465] LustreError: 353593:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9999.415366] LustreError: 353593:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [10002.424202] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10008.393206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10011.772243] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10016.583263] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10020.106341] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10024.875596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10025.185960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10025.190928] Lustre: Skipped 2 previous similar messages [10029.554665] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10035.016684] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10044.664597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10048.993664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10049.001080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.009441] Lustre: Skipped 1 previous similar message [10049.013883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10049.017688] Lustre: Skipped 5 previous similar messages [10050.419423] Lustre: server umount lustre-OST0000 complete [10059.201444] Lustre: server umount lustre-MDT0000 complete [10060.257391] LustreError: 353589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.270609] LustreError: 353589:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10061.539575] LustreError: 353574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955552 with bad export cookie 903911068293225419 [10061.545903] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10061.549568] LustreError: 353574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10061.732571] Lustre: server umount lustre-MDT0001 complete [10068.846296] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [10071.116076] Key type lgssc unregistered [10071.396309] LNet: 357142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10071.401639] LNetError: 357142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10072.422662] LNet: Removed LNI 192.168.206.126@tcp [10073.132163] Key type .llcrypt unregistered [10073.135305] Key type ._llcrypt unregistered [10091.835701] Key type ._llcrypt registered [10091.837687] Key type .llcrypt registered [10091.912326] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [10092.714413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10092.733207] alg: No test for adler32 (adler32-zlib) [10093.734562] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [10093.893471] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [10095.592198] Key type lgssc registered [10096.593774] Lustre: Echo OBD driver; http://www.lustre.org/ [10103.531401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10103.550801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10104.855692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10108.600240] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10115.043814] LustreError: 358655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10115.060965] LustreError: 358655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10115.631457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10115.953639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10119.454583] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10124.844505] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10128.567663] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10133.091801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10133.350493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10137.504840] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10142.841853] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10149.739764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10160.099199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10160.101715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10160.113692] Lustre: Skipped 2 previous similar messages [10160.121420] Lustre: Skipped 2 previous similar messages [10165.220143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10165.225385] Lustre: Skipped 2 previous similar messages [10165.649774] Lustre: server umount lustre-MDT0000 complete [10167.834709] LustreError: 358634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955658 with bad export cookie 16171469966312546181 [10167.842408] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10167.843333] LustreError: 358634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10168.023759] Lustre: server umount lustre-MDT0001 complete [10180.927905] Lustre: server umount lustre-OST0000 complete [10189.937363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10190.238375] LustreError: 362382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10190.297520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10193.423888] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10195.425639] LustreError: 362383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10199.532623] LustreError: 362382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10199.995539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10200.286529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10203.384884] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10209.286915] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10213.379599] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10218.096632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10218.344759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10222.224660] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10227.305752] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10234.737050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10250.213504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10250.216521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10250.227175] Lustre: Skipped 1 previous similar message [10250.236671] Lustre: Skipped 2 previous similar messages [10253.988348] Lustre: server umount lustre-MDT0000 complete [10255.330262] LustreError: 362378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.341944] LustreError: 362378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10256.682884] LustreError: 362362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955747 with bad export cookie 16171469966312547294 [10256.687927] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10256.704159] LustreError: 362362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10257.077051] Lustre: server umount lustre-MDT0001 complete [10271.270567] Lustre: server umount lustre-OST0000 complete [10285.989419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10286.331240] LustreError: 366312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10286.388698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10289.489516] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10295.535964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10299.224959] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10304.268427] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10308.463872] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10313.626979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10318.009755] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10323.144616] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10331.381972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10336.737653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10336.742103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10336.750208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10336.763959] Lustre: Skipped 2 previous similar messages [10336.774654] Lustre: Skipped 1 previous similar message [10338.446089] Lustre: server umount lustre-OST0000 complete [10341.859094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10341.860680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10341.870462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10341.886023] Lustre: Skipped 1 previous similar message [10346.773256] Lustre: server umount lustre-MDT0000 complete [10346.980757] LustreError: 367040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10346.993139] LustreError: 367040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10348.965787] LustreError: 368153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955839 with bad export cookie 16171469966312548316 [10348.969961] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10348.980900] LustreError: 368153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10349.208822] Lustre: server umount lustre-MDT0001 complete [10356.723585] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [10358.907151] Key type lgssc unregistered [10359.125334] LNet: 369860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10359.134970] LNetError: 369860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10359.149110] LNet: Removed LNI 192.168.206.126@tcp [10359.904205] Key type .llcrypt unregistered [10359.909206] Key type ._llcrypt unregistered [10375.961791] Key type ._llcrypt registered [10375.964387] Key type .llcrypt registered [10376.033445] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [10376.485223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10376.552882] alg: No test for adler32 (adler32-zlib) [10377.530226] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [10377.675977] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [10379.304356] Key type lgssc registered [10380.002833] Lustre: Echo OBD driver; http://www.lustre.org/ [10384.374000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10384.393471] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.400781] LustreError: 371296:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10384.411593] LustreError: 371296:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10384.415321] LustreError: 371296:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10384.419984] LustreError: 371296:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10384.426737] LustreError: 371296:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10389.025644] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10389.029712] LustreError: 371694:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10389.046799] LustreError: 371694:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10389.051712] LustreError: 371694:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10389.056129] LustreError: 371694:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10389.061319] LustreError: 371694:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10401.522299] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [10403.175412] Key type lgssc unregistered [10403.372557] LNet: 372695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10403.376891] LNetError: 372695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10403.392339] LNet: Removed LNI 192.168.206.126@tcp [10403.923581] Key type .llcrypt unregistered [10403.932110] Key type ._llcrypt unregistered [10414.675491] Key type ._llcrypt registered [10414.677459] Key type .llcrypt registered [10414.732206] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [10425.162226] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [10425.811712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10425.855480] alg: No test for adler32 (adler32-zlib) [10426.821644] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [10426.984634] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [10428.600321] Key type lgssc registered [10429.461484] Lustre: Echo OBD driver; http://www.lustre.org/ [10459.406128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10459.422206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10460.561314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10460.589165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10460.643454] Lustre: lustre-MDT0000: new disk, initializing [10460.700733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10460.715752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10463.862304] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10470.485122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10470.571125] Lustre: 375844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10470.604291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10470.607649] Lustre: Skipped 1 previous similar message [10470.654625] Lustre: lustre-MDT0001: new disk, initializing [10470.704365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10470.728064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10470.739239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10473.038837] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10479.198620] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10482.980298] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10487.060974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10487.261773] Lustre: lustre-OST0000: new disk, initializing [10487.264760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10487.270398] Lustre: 377808:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10487.348305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10487.491099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10487.496513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10487.600796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10491.294369] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10497.465275] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10500.526329] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10500.664250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10503.106238] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10503.254364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10508.266017] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.273613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10508.280434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10510.804415] Lustre: server umount lustre-OST0000 complete [10516.963322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10516.972706] LustreError: Skipped 1 previous similar message [10516.979739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10516.992376] Lustre: Skipped 1 previous similar message [10516.996087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10517.000823] Lustre: Skipped 1 previous similar message [10518.497948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10518.510389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10523.617426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10523.628782] Lustre: Skipped 2 previous similar messages [10528.224127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10528.330357] Lustre: server umount lustre-MDT0000 complete [10530.594495] LustreError: 375835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956021 with bad export cookie 15601100543079229533 [10530.602396] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10530.602543] LustreError: 375835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10530.851963] Lustre: server umount lustre-MDT0001 complete [10536.752680] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10537.848382] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:47:06 (1790956026) [10542.701227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10543.032708] LustreError: 380265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10543.043991] LustreError: 380265:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10543.108321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10546.350599] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10548.195191] LustreError: 380266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.142482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10552.375724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10554.941939] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10559.592818] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10563.164657] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10567.575777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10567.821077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10571.743275] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10576.495745] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10580.776316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10580.945820] Lustre: lustre-OST0001: new disk, initializing [10580.950591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10580.956993] Lustre: 383101:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10584.836381] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10590.751964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10590.758712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10590.834931] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10590.880924] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10595.809663] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10595.816341] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10595.820382] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10595.827334] Lustre: Skipped 1 previous similar message [10595.837313] Lustre: Skipped 1 previous similar message [10600.724156] Lustre: server umount lustre-OST0001 complete [10600.938574] LustreError: 383788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.951322] LustreError: 383788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10606.053549] LustreError: 383788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10606.061409] LustreError: 383788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10611.174663] LustreError: 382089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.183876] LustreError: 382089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10615.265449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10615.271485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.281342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10615.286525] Lustre: Skipped 1 previous similar message [10620.803548] Lustre: server umount lustre-OST0000 complete [10623.971387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10623.980244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.992300] Lustre: Skipped 1 previous similar message [10629.268365] Lustre: server umount lustre-MDT0000 complete [10631.650170] LustreError: 380993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10631.658306] LustreError: 380993:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10631.744961] LustreError: 380247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956122 with bad export cookie 15601100543079230415 [10631.746583] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10631.754873] LustreError: 380247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10637.996354] Lustre: server umount lustre-MDT0001 complete [10644.228579] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [10646.216448] Key type lgssc unregistered [10646.467358] LNet: 385088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10646.476641] LNetError: 385088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10646.491029] LNet: Removed LNI 192.168.206.126@tcp [10647.088865] Key type .llcrypt unregistered [10647.092710] Key type ._llcrypt unregistered [10678.290796] Key type ._llcrypt registered [10678.294407] Key type .llcrypt registered [10678.363970] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [10679.176250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10679.237146] alg: No test for adler32 (adler32-zlib) [10680.179760] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [10680.340754] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [10681.968219] Key type lgssc registered [10682.883470] Lustre: Echo OBD driver; http://www.lustre.org/ [10688.165602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10688.184635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.338902] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10689.359193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10689.484626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10691.859516] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10696.332600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10696.400706] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10696.420501] Lustre: 387878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10696.563093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10698.966914] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10703.660808] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10707.316839] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10712.038901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10712.047391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10712.059583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10713.572728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10713.574912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.689767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10718.703330] Lustre: Skipped 2 previous similar messages [10723.296123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10723.440876] Lustre: server umount lustre-MDT0000 complete [10726.196914] LustreError: 387869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956216 with bad export cookie 15451200689228898641 [10726.197540] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10726.207939] LustreError: 387869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10726.416318] Lustre: server umount lustre-MDT0001 complete [10733.687665] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:50:22 (1790956222) [10738.160348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.463104] LustreError: 390392:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10738.482240] LustreError: 390392:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10738.504860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10741.038771] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10743.777521] LustreError: 390393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.856235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10747.178198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10750.047950] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10755.896340] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10759.533918] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10764.737874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10764.861059] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10764.992347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10768.621129] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10773.208330] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10779.277222] Lustre: Permanently deactivating lustre-OST0000 [10779.290986] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10781.046578] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10782.109820] LustreError: 393230:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10782.125104] LustreError: 393230:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10782.131858] LustreError: 393230:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10783.824869] LustreError: 393279:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10783.840296] LustreError: 393279:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10791.366382] Lustre: server umount lustre-OST0000 complete [10793.554187] LustreError: 393528:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10793.558362] LustreError: 393528:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10797.543555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10797.554573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10797.554944] Lustre: Skipped 1 previous similar message [10797.582581] Lustre: Skipped 1 previous similar message [10802.657779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10802.662282] Lustre: Skipped 1 previous similar message [10805.806552] Lustre: server umount lustre-MDT0000 complete [10807.777486] LustreError: 390388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.790394] LustreError: 390388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10808.704548] LustreError: 390374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956299 with bad export cookie 15451200689228899208 [10808.709150] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10808.716937] LustreError: 390374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10819.131324] Lustre: server umount lustre-MDT0001 complete [10825.058099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10828.167274] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10829.863804] LustreError: 394686:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10834.250644] Lustre: 395025:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.126@tcp [10835.552244] Lustre: server umount MGS complete [10842.945451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10843.235843] LustreError: 395748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.254643] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10843.260912] Lustre: Skipped 1 previous similar message [10843.299737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10846.122829] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10848.745098] LustreError: 395749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.640880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.819365] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10854.092150] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10858.097548] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10861.176538] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10862.125405] Lustre: Permanently reactivating lustre-OST0000 [10862.137291] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10862.141855] Lustre: Skipped 2 previous similar messages [10891.232661] LustreError: 397626:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10896.890400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10897.112673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10897.117663] Lustre: Skipped 1 previous similar message [10901.260511] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10905.471534] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10916.834614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10916.837677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10916.854708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10916.854957] LustreError: Skipped 1 previous similar message [10916.867847] Lustre: Skipped 1 previous similar message [10920.382866] Lustre: server umount lustre-OST0000 complete [10923.490342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10923.497696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10923.509073] Lustre: Skipped 1 previous similar message [10929.707106] Lustre: server umount lustre-MDT0000 complete [10932.197346] LustreError: 395745:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.211501] LustreError: 395745:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10932.553306] LustreError: 397323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956423 with bad export cookie 15451200689228900258 [10932.561729] LustreError: 397323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10932.562517] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10932.835982] Lustre: server umount lustre-MDT0001 complete [10940.434969] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [10942.226746] Key type lgssc unregistered [10942.440737] LNet: 400352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10942.451781] LNetError: 400352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10942.466140] LNet: Removed LNI 192.168.206.126@tcp [10943.002250] Key type .llcrypt unregistered [10943.004086] Key type ._llcrypt unregistered [10954.626514] Key type ._llcrypt registered [10954.630651] Key type .llcrypt registered [10954.706961] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [10963.552519] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [10964.159657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10964.204041] alg: No test for adler32 (adler32-zlib) [10965.126687] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [10965.315261] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [10966.952212] Key type lgssc registered [10967.621698] Lustre: Echo OBD driver; http://www.lustre.org/ [10987.908391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10987.928845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.078271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10989.102460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10989.163780] Lustre: lustre-MDT0000: new disk, initializing [10989.210152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10989.223344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10991.946815] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10998.867608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.934967] Lustre: 403499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10998.978962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10998.982536] Lustre: Skipped 1 previous similar message [10999.026677] Lustre: lustre-MDT0001: new disk, initializing [10999.078848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10999.104177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10999.109775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11001.465325] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11007.126714] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11010.627789] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11015.218844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11015.388920] Lustre: lustre-OST0000: new disk, initializing [11015.392832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11015.398674] Lustre: 405465:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11015.460256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11020.311640] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11025.445496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11025.456689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11025.561904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11028.127200] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11031.609274] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11031.800827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11034.730106] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11034.913441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11036.131540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11036.132693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11036.145721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11036.151279] Lustre: Skipped 1 previous similar message [11041.259233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11041.270541] Lustre: Skipped 1 previous similar message [11042.171070] Lustre: server umount lustre-OST0000 complete [11045.351988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11045.360215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11045.368593] Lustre: Skipped 1 previous similar message [11045.383849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11046.370244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11051.286753] Lustre: server umount lustre-MDT0000 complete [11053.859169] LustreError: 404293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956544 with bad export cookie 6749156540861996641 [11053.865720] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11053.871014] LustreError: 404293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11054.179884] Lustre: server umount lustre-MDT0001 complete [11060.954214] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11062.187389] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:55:51 (1790956551) [11066.998915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11067.347974] LustreError: 407923:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11067.364839] LustreError: 407923:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11067.409304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11070.742783] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11072.483864] LustreError: 407924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11076.960668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.248825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11080.859583] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11086.079654] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11090.172724] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11094.811802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11095.077374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11098.971892] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11103.933528] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11105.252879] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11112.744104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11117.541163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11117.544772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11117.554213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11117.558350] Lustre: Skipped 3 previous similar messages [11121.921311] Lustre: server umount lustre-OST0000 complete [11127.780337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11127.790218] Lustre: Skipped 1 previous similar message [11127.795911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11127.799382] Lustre: Skipped 1 previous similar message [11138.018888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11138.024189] Lustre: Skipped 3 previous similar messages [11139.040138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11139.125694] Lustre: server umount lustre-MDT0000 complete [11141.227673] LustreError: 407905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956631 with bad export cookie 6749156540861997523 [11141.230920] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11141.240866] LustreError: 407905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11141.442290] Lustre: server umount lustre-MDT0001 complete [11147.288101] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11148.832839] Key type lgssc unregistered [11149.035755] LNet: 411646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11149.041621] LNetError: 411646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11149.058830] LNet: Removed LNI 192.168.206.126@tcp [11149.561988] Key type .llcrypt unregistered [11149.564526] Key type ._llcrypt unregistered [11165.429462] Key type ._llcrypt registered [11165.431863] Key type .llcrypt registered [11165.531605] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11165.762495] Key type .llcrypt unregistered [11165.765574] Key type ._llcrypt unregistered [11177.059463] Key type ._llcrypt registered [11177.061163] Key type .llcrypt registered [11177.135460] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [11177.768143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11177.787603] alg: No test for adler32 (adler32-zlib) [11178.735412] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [11178.904696] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [11180.537414] Key type lgssc registered [11181.388113] Lustre: Echo OBD driver; http://www.lustre.org/ [11186.752417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11186.782472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11188.069893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11191.217399] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11196.932903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11197.162900] LustreError: 413849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11197.185947] LustreError: 413849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11197.198726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11198.178545] LustreError: 413848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.189381] LustreError: 413848:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11198.194662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11201.342849] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11203.555595] LustreError: 413843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.574605] LustreError: 413843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11206.103481] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11208.677519] LustreError: 413844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11208.693250] LustreError: 413844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11210.685150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11210.956544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11213.448680] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11224.033608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11224.041142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.050927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11226.595519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11226.595523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.599359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11226.599456] Lustre: Skipped 1 previous similar message [11228.092889] Lustre: server umount lustre-OST0000 complete [11231.717807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11231.718718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11231.723702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11231.724475] LustreError: Skipped 1 previous similar message [11231.738987] Lustre: Skipped 1 previous similar message [11232.513464] Lustre: server umount lustre-MDT0000 complete [11234.765361] LustreError: 413830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956725 with bad export cookie 12757275600365459583 [11234.770369] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11234.776528] LustreError: 413830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11234.961440] Lustre: server umount lustre-MDT0001 complete [11241.001066] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11242.605419] Key type lgssc unregistered [11242.786995] LNet: 416994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11242.792511] LNetError: 416994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11242.811887] LNet: Removed LNI 192.168.206.126@tcp [11243.275217] Key type .llcrypt unregistered [11243.277223] Key type ._llcrypt unregistered [11256.368348] Key type ._llcrypt registered [11256.370103] Key type .llcrypt registered [11256.433953] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [11256.941979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11256.970333] alg: No test for adler32 (adler32-zlib) [11257.868860] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [11258.002497] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [11259.608422] Key type lgssc registered [11260.481419] Lustre: Echo OBD driver; http://www.lustre.org/ [11264.619437] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11264.631597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11280.096370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11285.216433] LustreError: 418377:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.126@tcp: failed processing log, type 4: rc = -110 [11311.840220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11314.770341] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11318.617481] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11322.031157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.285366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11322.318486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11324.109957] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11328.048952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11328.261939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11330.228629] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11333.743512] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11336.280149] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11342.817268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11342.821321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11342.828447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11343.844813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11343.848731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11343.849794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11343.858474] Lustre: Skipped 1 previous similar message [11345.322741] Lustre: server umount lustre-OST0000 complete [11348.962214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11348.962280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11348.963120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11348.974901] Lustre: Skipped 1 previous similar message [11353.123996] Lustre: server umount lustre-MDT0000 complete [11354.084756] LustreError: 419414:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11354.094910] LustreError: 419414:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11354.706419] LustreError: 418383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956845 with bad export cookie 15535525440039588753 [11354.709756] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11354.713110] LustreError: 418383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11354.855963] Lustre: server umount lustre-MDT0001 complete [11359.218975] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11360.459773] Key type lgssc unregistered [11360.609487] LNet: 421942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11360.613822] LNetError: 421942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11360.621624] LNet: Removed LNI 192.168.206.126@tcp [11361.015456] Key type .llcrypt unregistered [11361.017166] Key type ._llcrypt unregistered [11371.656968] Key type ._llcrypt registered [11371.658868] Key type .llcrypt registered [11371.710366] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [11372.087334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11372.139916] alg: No test for adler32 (adler32-zlib) [11373.039755] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [11373.146409] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [11374.744199] Key type lgssc registered [11375.236757] Lustre: Echo OBD driver; http://www.lustre.org/ [11378.562162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11378.572079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11379.774819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11381.371740] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11385.107303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11385.305188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11387.077514] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11390.236210] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11392.888670] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11395.856697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11396.081795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11398.617382] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11401.795404] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11405.298945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11410.338718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.340849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11410.347777] Lustre: Skipped 2 previous similar messages [11410.358614] Lustre: Skipped 2 previous similar messages [11413.584595] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.26@tcp (stopping) [11415.494415] Lustre: server umount lustre-MDT0000 complete [11415.522839] LustreError: 424599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11415.530645] LustreError: 424599:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11416.528928] LustreError: 425936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.26@tcp arrived at 1790956907 with bad export cookie 16266199607817624563 [11416.538516] LustreError: 425936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11417.313765] LustreError: 423343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956907 with bad export cookie 16266199607817623905 [11417.317247] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11417.320170] LustreError: 423343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11420.640971] LustreError: 423361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11420.641500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11420.647732] LustreError: 423361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11420.654039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11422.689103] LustreError: 423362:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.692199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11422.698598] LustreError: 423362:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11423.464195] Lustre: server umount lustre-OST0000 complete [11426.680278] LustreError: 424090:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000006628378 x1877954383603072/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11426.753478] Lustre: server umount lustre-MDT0001 complete [11431.038787] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11432.218042] Key type lgssc unregistered [11432.361550] LNet: 427042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11432.368682] LNetError: 427042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11432.379356] LNet: Removed LNI 192.168.206.126@tcp [11432.743651] Key type .llcrypt unregistered [11432.745225] Key type ._llcrypt unregistered [11443.952730] Key type ._llcrypt registered [11443.954464] Key type .llcrypt registered [11444.001675] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [11444.412227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11444.457271] alg: No test for adler32 (adler32-zlib) [11445.353187] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [11445.457384] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [11447.056221] Key type lgssc registered [11447.567413] Lustre: Echo OBD driver; http://www.lustre.org/ [11451.014862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11451.028799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11452.240789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11453.899315] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11457.255076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11457.422707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11459.146296] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11462.154192] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11464.612081] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11467.695915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11467.860432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11470.195142] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11473.187877] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11477.095658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11482.081863] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11482.084916] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11482.091029] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11482.597127] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11482.603992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11484.750448] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.26@tcp (stopping) [11486.678616] Lustre: server umount lustre-MDT0001 complete [11487.719928] LustreError: 429187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11487.732170] LustreError: 429187:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11492.833822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11492.834838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11492.835041] LustreError: 428450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11492.852540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11494.872296] Lustre: server umount lustre-OST0000 complete [11496.524610] Lustre: server umount lustre-MDT0000 complete [11502.104728] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11503.270111] Key type lgssc unregistered [11503.415837] LNet: 432128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11503.419717] LNetError: 432128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11503.433341] LNet: Removed LNI 192.168.206.126@tcp [11503.803957] Key type .llcrypt unregistered [11503.805603] Key type ._llcrypt unregistered [11513.658895] Key type ._llcrypt registered [11513.660670] Key type .llcrypt registered [11513.705142] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [11520.762439] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [11521.095205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11521.115790] alg: No test for adler32 (adler32-zlib) [11522.010415] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [11522.114312] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [11523.720270] Key type lgssc registered [11524.245326] Lustre: Echo OBD driver; http://www.lustre.org/ [11541.025092] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [11547.929311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11547.941438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11549.042231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11549.058650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11549.107326] Lustre: lustre-MDT0000: new disk, initializing [11549.143644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11549.153478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11550.925355] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11558.056245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11558.177685] Lustre: lustre-OST0000: new disk, initializing [11558.180316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11558.183828] Lustre: Skipped 1 previous similar message [11558.187858] Lustre: 437148:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11558.221801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11560.595540] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11564.726895] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11566.084410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11566.088564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11566.100413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11576.004361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11576.054153] Lustre: 436129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11576.087576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11576.131809] Lustre: lustre-MDT0001: new disk, initializing [11576.187695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11576.204787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11576.209293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11577.609420] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11581.400218] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11581.489486] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11583.034842] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11583.125476] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11587.042062] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11587.046789] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11587.055993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11587.061641] Lustre: Skipped 1 previous similar message [11589.717441] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.26@tcp (stopping) [11592.163375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11592.166067] Lustre: Skipped 1 previous similar message [11594.824042] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.26@tcp (stopping) [11599.328485] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11599.417308] Lustre: server umount lustre-MDT0001 complete [11607.521419] LustreError: 439723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11607.527755] LustreError: 439723:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11612.129718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11612.134401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11612.139734] Lustre: Skipped 1 previous similar message [11612.142419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11612.145148] Lustre: Skipped 2 previous similar messages [11612.641073] LustreError: 436142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11617.761868] LustreError: 438489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.206837] Lustre: server umount lustre-OST0000 complete [11634.144126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11634.247942] Lustre: server umount lustre-MDT0000 complete [11639.032453] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11640.074506] Key type lgssc unregistered [11640.201437] LNet: 440896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11640.204174] LNetError: 440896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11640.216319] LNet: Removed LNI 192.168.206.126@tcp [11640.521509] Key type .llcrypt unregistered [11640.522941] Key type ._llcrypt unregistered [11654.152727] Key type ._llcrypt registered [11654.154341] Key type .llcrypt registered [11654.196338] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11654.306356] Key type .llcrypt unregistered [11654.308300] Key type ._llcrypt unregistered [11661.429046] Key type ._llcrypt registered [11661.430811] Key type .llcrypt registered [11661.476585] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [11661.922053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11661.959657] alg: No test for adler32 (adler32-zlib) [11662.832545] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [11662.933595] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [11664.520165] Key type lgssc registered [11664.981419] Lustre: Echo OBD driver; http://www.lustre.org/ [11676.033903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11676.048372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11677.144994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11677.159323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11677.200074] Lustre: lustre-MDT0000: new disk, initializing [11677.225220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11677.232469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11678.788173] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11683.738860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.790841] Lustre: 444258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11683.819877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11683.823294] Lustre: Skipped 1 previous similar message [11683.870114] Lustre: lustre-MDT0001: new disk, initializing [11683.913244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11683.931374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11683.936125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11685.532104] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11689.747853] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11691.842160] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11694.438176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11694.558092] Lustre: lustre-OST0000: new disk, initializing [11694.560901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11694.565059] Lustre: 446224:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11694.609987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11696.930855] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11701.141413] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11703.804194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11703.809502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11703.840818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11708.898358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11708.899884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11708.904144] Lustre: Skipped 1 previous similar message [11714.019300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11714.021815] Lustre: Skipped 4 previous similar messages [11714.929775] Lustre: server umount lustre-MDT0000 complete [11716.429925] LustreError: 444250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957207 with bad export cookie 7916991860980365357 [11716.434656] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11716.436092] LustreError: 444250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11716.567068] Lustre: server umount lustre-MDT0001 complete [11728.290532] Lustre: server umount lustre-OST0000 complete [11732.448752] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:07:01 (1790957221) [11735.489431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11735.695288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11737.259298] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11740.514493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11740.651746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11742.192907] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11744.945045] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11747.167524] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11762.851642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11762.935175] Lustre: Found index 0 for lustre-OST0000, updating log [11763.006114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11765.160599] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11767.835517] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11781.479200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11786.726903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.727413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11786.729626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11786.732150] Lustre: Skipped 2 previous similar messages [11786.736580] LustreError: Skipped 1 previous similar message [11787.174223] Lustre: server umount lustre-OST0000 complete [11800.013111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.013542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11800.019417] Lustre: Skipped 1 previous similar message [11800.023780] Lustre: Skipped 2 previous similar messages [11804.129405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11804.132228] Lustre: Skipped 3 previous similar messages [11814.368206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11814.369213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11814.374873] Lustre: Skipped 3 previous similar messages [11814.432069] Lustre: server umount lustre-MDT0000 complete [11815.760166] LustreError: 450092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957306 with bad export cookie 7916991860980366736 [11815.763066] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11815.765258] LustreError: 450092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11815.881740] Lustre: server umount lustre-MDT0001 complete [11818.897982] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:08:28 (1790957308) [11823.712235] LNet: 1 local NIs in recovery (showing 1): 192.168.206.126@tcp [12399.375212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12399.539500] LustreError: 454674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12399.546231] LustreError: 454674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12399.566833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12400.958439] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12403.763855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12403.886254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12405.362942] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12407.817647] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12409.628693] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12424.931201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12425.012246] Lustre: Found index 0 for lustre-OST0000, updating log [12425.099654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12426.926510] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12428.198535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12429.188924] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12430.818066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12430.823790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12435.918028] Lustre: server umount lustre-OST0000 complete [12438.177568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.181186] Lustre: Skipped 1 previous similar message [12443.292884] Lustre: server umount lustre-MDT0000 complete [12443.298946] LustreError: 455402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12443.306443] LustreError: 455402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12444.321378] LustreError: 454675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12444.328522] LustreError: 454675:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12444.526284] LustreError: 456576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957935 with bad export cookie 7916991860980367807 [12444.530420] LustreError: 456576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12444.530903] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12444.650436] Lustre: server umount lustre-MDT0001 complete [12451.424192] LNet: 1 local NIs in recovery (showing 1): 192.168.206.126@tcp [12461.297244] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [12464.261588] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12475.408309] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:19:25 (1790957965) [12481.038255] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [12482.023650] Key type lgssc unregistered [12487.264352] LNet: 461325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12487.266409] LNetError: 461325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12555.878339] LNet: Removed LNI 192.168.206.126@tcp [12556.168915] Key type .llcrypt unregistered [12556.170442] Key type ._llcrypt unregistered [12562.556766] Key type ._llcrypt registered [12562.557664] Key type .llcrypt registered [12562.588514] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12562.864795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12562.910130] alg: No test for adler32 (adler32-zlib) [12563.759058] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [12563.845525] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [12565.424153] Key type lgssc registered [12565.759385] Lustre: Echo OBD driver; http://www.lustre.org/ [12567.024186] LNet: Added LNI 192.168.206.126@tcp100 [8/256/0/180] [12567.289570] LNet: Added LNI 192.168.206.126@tcp101 [8/256/0/180] [12567.562901] LNet: Added LNI 192.168.206.126@tcp102 [8/256/0/180] [12567.844254] LNet: Added LNI 192.168.206.126@tcp103 [8/256/0/180] [12568.111291] LNet: Added LNI 192.168.206.126@tcp104 [8/256/0/180] [12568.385399] LNet: Added LNI 192.168.206.126@tcp105 [8/256/0/180] [12568.650741] LNet: Added LNI 192.168.206.126@tcp106 [8/256/0/180] [12568.921294] LNet: Added LNI 192.168.206.126@tcp107 [8/256/0/180] [12569.188624] LNet: Added LNI 192.168.206.126@tcp108 [8/256/0/180] [12569.463245] LNet: Added LNI 192.168.206.126@tcp109 [8/256/0/180] [12569.730496] LNet: Added LNI 192.168.206.126@tcp110 [8/256/0/180] [12569.993015] LNet: Added LNI 192.168.206.126@tcp111 [8/256/0/180] [12570.265878] LNet: Added LNI 192.168.206.126@tcp112 [8/256/0/180] [12570.538188] LNet: Added LNI 192.168.206.126@tcp113 [8/256/0/180] [12570.800988] LNet: Added LNI 192.168.206.126@tcp114 [8/256/0/180] [12571.065537] LNet: Added LNI 192.168.206.126@tcp115 [8/256/0/180] [12571.344655] LNet: Added LNI 192.168.206.126@tcp116 [8/256/0/180] [12571.618947] LNet: Added LNI 192.168.206.126@tcp117 [8/256/0/180] [12571.891555] LNet: Added LNI 192.168.206.126@tcp118 [8/256/0/180] [12572.157335] LNet: Added LNI 192.168.206.126@tcp119 [8/256/0/180] [12572.428690] LNet: Added LNI 192.168.206.126@tcp120 [8/256/0/180] [12572.697847] LNet: Added LNI 192.168.206.126@tcp121 [8/256/0/180] [12572.972247] LNet: Added LNI 192.168.206.126@tcp122 [8/256/0/180] [12573.250152] LNet: Added LNI 192.168.206.126@tcp123 [8/256/0/180] [12573.515239] LNet: Added LNI 192.168.206.126@tcp124 [8/256/0/180] [12573.780233] LNet: Added LNI 192.168.206.126@tcp125 [8/256/0/180] [12574.056239] LNet: Added LNI 192.168.206.126@tcp126 [8/256/0/180] [12574.333815] LNet: Added LNI 192.168.206.126@tcp127 [8/256/0/180] [12574.604180] LNet: Added LNI 192.168.206.126@tcp128 [8/256/0/180] [12574.868848] LNet: Added LNI 192.168.206.126@tcp129 [8/256/0/180] [12575.131354] LNet: Added LNI 192.168.206.126@tcp130 [8/256/0/180] [12575.393037] LNet: Added LNI 192.168.206.126@tcp131 [8/256/0/180] [12575.665094] LNet: Added LNI 192.168.206.126@tcp132 [8/256/0/180] [12575.942804] LNet: Added LNI 192.168.206.126@tcp133 [8/256/0/180] [12576.201919] LNet: Added LNI 192.168.206.126@tcp134 [8/256/0/180] [12576.481857] LNet: Added LNI 192.168.206.126@tcp135 [8/256/0/180] [12578.581341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12578.596738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.632813] Lustre: 464338:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.126@tcp: can't add NID 192.168.206.126@tcp130: rc = -75 [12579.655403] LNetError: 461792:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.126 [12579.659381] LNetError: Protocol error connecting to 192.168.206.126@tcp118 at host 192.168.206.126:988: is it running a compatible version of Lustre? [12579.663938] LNetError: Skipped 3 previous similar messages [12579.714091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12579.723684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12579.757755] Lustre: lustre-MDT0000: new disk, initializing [12579.775169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12579.780018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12580.256234] LNet: 6 local NIs in recovery (showing 6): 192.168.206.126@tcp123, 192.168.206.126@tcp118, 192.168.206.126@tcp125, 192.168.206.126@tcp128, 192.168.206.126@tcp129, 192.168.206.126@tcp127 [12580.258044] LNetError: 464360:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.126 [12580.262914] LNetError: 464360:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [12580.265021] LNetError: Protocol error connecting to 192.168.206.126@tcp123 at host 192.168.206.126:988: is it running a compatible version of Lustre? [12580.268127] LNetError: Skipped 8 previous similar messages [12581.049897] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12582.305038] LNetError: 461791:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.126 [12582.307772] LNetError: 461791:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [12582.310062] LNetError: Protocol error connecting to 192.168.206.126@tcp123 at host 192.168.206.126:988: is it running a compatible version of Lustre? [12584.996975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12585.048431] Lustre: 464354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12585.075652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12585.078129] Lustre: Skipped 1 previous similar message [12585.114151] Lustre: lustre-MDT0001: new disk, initializing [12585.125233] Lustre: 465158:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.126@tcp: can't add NID 192.168.206.126@tcp130: rc = -75 [12585.127728] Lustre: 465158:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12585.168950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12585.182094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12585.186038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12586.401389] LNetError: 464366:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.126 [12586.405836] LNetError: Protocol error connecting to 192.168.206.126@tcp123 at host 192.168.206.126:988: is it running a compatible version of Lustre? [12586.425323] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12589.554448] Lustre: 465729:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.126@tcp: can't add NID 192.168.206.126@tcp130: rc = -75 [12589.557047] Lustre: 465729:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12589.965156] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12591.711779] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12593.779142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12593.884624] Lustre: lustre-OST0000: new disk, initializing [12593.886602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12593.890058] Lustre: 466331:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12593.918370] Lustre: 466336:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.126@tcp: can't add NID 192.168.206.126@tcp130: rc = -75 [12593.920862] Lustre: 466336:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12593.936104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12594.593073] LNetError: 464364:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.126 [12594.595896] LNetError: Protocol error connecting to 192.168.206.126@tcp123 at host 192.168.206.126:988: is it running a compatible version of Lustre? [12595.645274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12595.650090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12595.689097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12595.707754] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12599.280492] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12605.922220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12605.922377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12605.924558] LustreError: Skipped 1 previous similar message [12605.925092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12605.927659] Lustre: Skipped 1 previous similar message [12605.934047] Lustre: Skipped 1 previous similar message [12607.633373] Lustre: server umount lustre-OST0000 complete [12609.953151] LNetError: 461790:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.126 [12609.955799] LNetError: Protocol error connecting to 192.168.206.126@tcp115 at host 192.168.206.126:988: is it running a compatible version of Lustre? [12611.043385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12611.043508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12611.046482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12611.051226] Lustre: Skipped 1 previous similar message [12616.161130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12616.163887] Lustre: Skipped 2 previous similar messages [12621.281082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12621.283940] Lustre: Skipped 1 previous similar message [12623.328102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12623.387522] Lustre: server umount lustre-MDT0000 complete [12624.548146] LustreError: 464348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958115 with bad export cookie 13572184279249745762 [12624.550155] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12624.552920] LustreError: 464348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12624.668369] Lustre: server umount lustre-MDT0001 complete [12628.111130] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [12628.385045] LNetError: 464366:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.126 [12628.388255] LNetError: 464366:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [12628.390827] LNetError: Protocol error connecting to 192.168.206.126@tcp129 at host 192.168.206.126:988: is it running a compatible version of Lustre? [12628.395532] LNetError: Skipped 8 previous similar messages [12629.096969] Key type lgssc unregistered [12629.248707] LNet: 468242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12629.251499] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12629.253567] LNet: Removed LNI 192.168.206.126@tcp [12630.304151] LNet: Removed LNI 192.168.206.126@tcp100 [12630.305448] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12630.307632] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12630.310710] LNet: Removed LNI 192.168.206.126@tcp101 [12631.328130] LNet: Removed LNI 192.168.206.126@tcp102 [12631.329837] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12631.331710] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12631.333793] LNet: Removed LNI 192.168.206.126@tcp103 [12632.352142] LNet: Removed LNI 192.168.206.126@tcp104 [12633.376115] LNet: Removed LNI 192.168.206.126@tcp105 [12633.377345] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12633.379358] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12633.381400] LNet: Removed LNI 192.168.206.126@tcp106 [12634.400139] LNet: Removed LNI 192.168.206.126@tcp107 [12635.424111] LNet: Removed LNI 192.168.206.126@tcp108 [12636.448161] LNet: Removed LNI 192.168.206.126@tcp109 [12637.472128] LNet: Removed LNI 192.168.206.126@tcp110 [12637.473440] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12637.476185] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12637.478622] LNet: Removed LNI 192.168.206.126@tcp111 [12638.496164] LNet: Removed LNI 192.168.206.126@tcp112 [12639.520112] LNet: Removed LNI 192.168.206.126@tcp113 [12640.544136] LNet: Removed LNI 192.168.206.126@tcp114 [12641.568164] LNet: Removed LNI 192.168.206.126@tcp115 [12642.592132] LNet: Removed LNI 192.168.206.126@tcp116 [12643.616181] LNet: Removed LNI 192.168.206.126@tcp117 [12644.640148] LNet: Removed LNI 192.168.206.126@tcp118 [12645.664178] LNet: Removed LNI 192.168.206.126@tcp119 [12645.666592] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12645.669299] LNetError: 468242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12645.673779] LNet: Removed LNI 192.168.206.126@tcp120 [12646.688129] LNet: Removed LNI 192.168.206.126@tcp121 [12647.712182] LNet: Removed LNI 192.168.206.126@tcp122 [12647.714264] LNet: Removed LNI 192.168.206.126@tcp123 [12648.736143] LNet: Removed LNI 192.168.206.126@tcp124 [12649.760118] LNet: Removed LNI 192.168.206.126@tcp125 [12650.784119] LNet: Removed LNI 192.168.206.126@tcp126 [12651.808133] LNet: Removed LNI 192.168.206.126@tcp127 [12652.832140] LNet: Removed LNI 192.168.206.126@tcp128 [12653.856154] LNet: Removed LNI 192.168.206.126@tcp129 [12653.857855] LNet: Removed LNI 192.168.206.126@tcp130 [12653.859222] LNet: Removed LNI 192.168.206.126@tcp131 [12653.860631] LNet: Removed LNI 192.168.206.126@tcp132 [12653.861804] LNet: Removed LNI 192.168.206.126@tcp133 [12653.863358] LNet: Removed LNI 192.168.206.126@tcp134 [12653.870428] LNet: Removed LNI 192.168.206.126@tcp135 [12654.114570] Key type .llcrypt unregistered [12654.115803] Key type ._llcrypt unregistered [12660.874385] Key type ._llcrypt registered [12660.875454] Key type .llcrypt registered [12660.907900] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [12666.783196] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12667.071967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12667.111065] alg: No test for adler32 (adler32-zlib) [12667.948028] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [12668.024860] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [12669.600171] Key type lgssc registered [12669.952840] Lustre: Echo OBD driver; http://www.lustre.org/ [12680.481096] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:22:50 (1790958170) [12685.788951] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [12686.710385] Key type lgssc unregistered [12686.832277] LNet: 472313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12686.834350] LNetError: 472313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12686.842196] LNet: Removed LNI 192.168.206.126@tcp [12687.106184] Key type .llcrypt unregistered [12687.107196] Key type ._llcrypt unregistered [12693.222913] Key type ._llcrypt registered [12693.224137] Key type .llcrypt registered [12693.256456] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12693.561426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12693.597459] alg: No test for adler32 (adler32-zlib) [12694.457861] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [12694.537805] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [12696.112185] Key type lgssc registered [12696.458209] Lustre: Echo OBD driver; http://www.lustre.org/ [12699.354878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12699.360967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12700.439863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12700.450863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12700.486076] Lustre: lustre-MDT0000: new disk, initializing [12700.504934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12700.510676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12701.740687] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12705.575902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12705.615290] Lustre: 473520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12705.638547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12705.640372] Lustre: Skipped 1 previous similar message [12705.674644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12705.679710] Lustre: lustre-MDT0001: new disk, initializing [12705.708517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12705.720862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12705.723901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12706.929104] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12710.523259] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12712.301343] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12714.478868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12714.570165] Lustre: lustre-OST0000: new disk, initializing [12714.572411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12714.575700] Lustre: 475484:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12714.603511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12715.575964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12715.578898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12715.611123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12716.390281] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12719.875994] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12720.964405] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing lnet_if_list [12722.537606] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12722.549890] LNet: Added LNI 192.168.206.126@tcp10 [8/256/0/180] [12724.327319] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12724.339903] LNet: Added LNI 192.168.206.126@tcp11 [8/256/0/180] [12725.716616] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12725.732429] LNet: Added LNI 192.168.206.126@tcp12 [8/256/0/180] [12727.151930] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12727.166461] LNet: Added LNI 192.168.206.126@tcp13 [8/256/0/180] [12728.589368] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12728.601186] LNet: Added LNI 192.168.206.126@tcp14 [8/256/0/180] [12730.049431] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12730.061788] LNet: Added LNI 192.168.206.126@tcp15 [8/256/0/180] [12731.555085] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12731.569402] LNet: Added LNI 192.168.206.126@tcp16 [8/256/0/180] [12733.097986] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12733.111653] LNet: Added LNI 192.168.206.126@tcp17 [8/256/0/180] [12734.715271] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12734.729504] LNet: Added LNI 192.168.206.126@tcp18 [8/256/0/180] [12736.362255] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12736.374839] LNet: Added LNI 192.168.206.126@tcp19 [8/256/0/180] [12737.949619] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12737.961787] LNet: Added LNI 192.168.206.126@tcp20 [8/256/0/180] [12739.671642] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12739.685516] LNet: Added LNI 192.168.206.126@tcp21 [8/256/0/180] [12741.417577] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12741.430469] LNet: Added LNI 192.168.206.126@tcp22 [8/256/0/180] [12743.206052] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12743.220083] LNet: Added LNI 192.168.206.126@tcp23 [8/256/0/180] [12744.829668] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12744.841442] LNet: Added LNI 192.168.206.126@tcp24 [8/256/0/180] [12746.667108] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12746.678107] LNet: Added LNI 192.168.206.126@tcp25 [8/256/0/180] [12748.601524] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12748.614278] LNet: Added LNI 192.168.206.126@tcp26 [8/256/0/180] [12750.497353] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12750.510033] LNet: Added LNI 192.168.206.126@tcp27 [8/256/0/180] [12752.401065] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12752.414084] LNet: Added LNI 192.168.206.126@tcp28 [8/256/0/180] [12754.350213] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12754.364972] LNet: Added LNI 192.168.206.126@tcp29 [8/256/0/180] [12756.413698] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12756.427029] LNet: Added LNI 192.168.206.126@tcp30 [8/256/0/180] [12758.399742] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12758.412905] LNet: Added LNI 192.168.206.126@tcp31 [8/256/0/180] [12760.349268] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12760.360796] LNet: Added LNI 192.168.206.126@tcp32 [8/256/0/180] [12762.508639] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12762.525864] LNet: Added LNI 192.168.206.126@tcp33 [8/256/0/180] [12764.617786] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12764.631252] LNet: Added LNI 192.168.206.126@tcp34 [8/256/0/180] [12766.761989] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12766.774222] LNet: Added LNI 192.168.206.126@tcp35 [8/256/0/180] [12769.144071] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12769.156879] LNet: Added LNI 192.168.206.126@tcp36 [8/256/0/180] [12771.461214] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12771.475529] LNet: Added LNI 192.168.206.126@tcp37 [8/256/0/180] [12773.800980] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12773.814453] LNet: Added LNI 192.168.206.126@tcp38 [8/256/0/180] [12776.171380] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12776.186621] LNet: Added LNI 192.168.206.126@tcp39 [8/256/0/180] [12778.562134] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12778.574771] LNet: Added LNI 192.168.206.126@tcp40 [8/256/0/180] [12780.897662] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12780.908595] LNet: Added LNI 192.168.206.126@tcp41 [8/256/0/180] [12783.282381] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12783.297030] LNet: Added LNI 192.168.206.126@tcp42 [8/256/0/180] [12785.803507] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12785.814619] LNet: Added LNI 192.168.206.126@tcp43 [8/256/0/180] [12788.254738] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12788.267948] LNet: Added LNI 192.168.206.126@tcp44 [8/256/0/180] [12790.743906] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12790.757735] LNet: Added LNI 192.168.206.126@tcp45 [8/256/0/180] [12793.374244] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12793.388280] LNet: Added LNI 192.168.206.126@tcp46 [8/256/0/180] [12796.082132] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12796.095642] LNet: Added LNI 192.168.206.126@tcp47 [8/256/0/180] [12798.911393] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12798.923625] LNet: Added LNI 192.168.206.126@tcp48 [8/256/0/180] [12801.610844] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12801.621567] LNet: Added LNI 192.168.206.126@tcp49 [8/256/0/180] [12804.289942] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12804.302156] LNet: Added LNI 192.168.206.126@tcp50 [8/256/0/180] [12812.769655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12812.772151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12812.774362] Lustre: Skipped 2 previous similar messages [12812.777579] Lustre: Skipped 2 previous similar messages [12817.754234] Lustre: server umount lustre-MDT0000 complete [12818.897195] LustreError: 473511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958309 with bad export cookie 2444683166677661433 [12818.899982] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12818.901708] LustreError: 473511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12819.001560] Lustre: server umount lustre-MDT0001 complete [12825.864884] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12828.732317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12829.152319] LustreError: 472779:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cb54ad84 x1877955769877120/t0(0) o250->MGC192.168.206.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12829.240366] LustreError: 475478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12829.247065] LustreError: 475478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12829.252328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12830.549467] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12833.270862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12833.355916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12833.396716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12834.735928] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12837.114825] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12838.480121] Lustre: 472783:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790958313/real 1790958313] req@00000000516e46a7 x1877955769876864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790958329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12838.884816] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12838.889866] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12842.981210] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12844.512087] Lustre: 472780:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790958318/real 1790958318] req@000000003a12ed07 x1877955769876992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790958334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12848.098435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12848.099770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.102549] Lustre: Skipped 1 previous similar message [12851.859988] Lustre: server umount lustre-OST0000 complete [12853.217318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12853.217728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12853.221761] Lustre: Skipped 1 previous similar message [12853.224590] Lustre: Skipped 1 previous similar message [12858.337583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12858.339289] Lustre: Skipped 1 previous similar message [12859.156642] Lustre: server umount lustre-MDT0000 complete [12860.276271] LustreError: 473512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958350 with bad export cookie 2444683166677663498 [12860.278515] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12860.282140] LustreError: 473512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12860.396527] Lustre: server umount lustre-MDT0001 complete [12863.655721] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [12864.626082] Key type lgssc unregistered [12864.773654] LNet: 502441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12864.776569] LNetError: 502441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12864.778965] LNet: Removed LNI 192.168.206.126@tcp [12864.780281] LNet: Removed LNI 192.168.206.126@tcp10 [12864.781652] LNet: Removed LNI 192.168.206.126@tcp11 [12864.782790] LNet: Removed LNI 192.168.206.126@tcp12 [12864.784072] LNet: Removed LNI 192.168.206.126@tcp13 [12864.785206] LNet: Removed LNI 192.168.206.126@tcp14 [12864.786398] LNet: Removed LNI 192.168.206.126@tcp15 [12864.787695] LNet: Removed LNI 192.168.206.126@tcp16 [12864.789084] LNet: Removed LNI 192.168.206.126@tcp17 [12864.790280] LNet: Removed LNI 192.168.206.126@tcp18 [12864.791422] LNet: Removed LNI 192.168.206.126@tcp19 [12864.792715] LNet: Removed LNI 192.168.206.126@tcp20 [12864.794084] LNet: Removed LNI 192.168.206.126@tcp21 [12864.795196] LNet: Removed LNI 192.168.206.126@tcp22 [12864.796344] LNet: Removed LNI 192.168.206.126@tcp23 [12864.797533] LNet: Removed LNI 192.168.206.126@tcp24 [12864.798780] LNet: Removed LNI 192.168.206.126@tcp25 [12864.799959] LNet: Removed LNI 192.168.206.126@tcp26 [12864.801082] LNet: Removed LNI 192.168.206.126@tcp27 [12864.802386] LNet: Removed LNI 192.168.206.126@tcp28 [12864.803523] LNet: Removed LNI 192.168.206.126@tcp29 [12864.804725] LNet: Removed LNI 192.168.206.126@tcp30 [12864.805908] LNet: Removed LNI 192.168.206.126@tcp31 [12864.807055] LNet: Removed LNI 192.168.206.126@tcp32 [12864.808281] LNet: Removed LNI 192.168.206.126@tcp33 [12864.809454] LNet: Removed LNI 192.168.206.126@tcp34 [12864.810539] LNet: Removed LNI 192.168.206.126@tcp35 [12864.811694] LNet: Removed LNI 192.168.206.126@tcp36 [12864.812942] LNet: Removed LNI 192.168.206.126@tcp37 [12864.814355] LNet: Removed LNI 192.168.206.126@tcp38 [12864.815451] LNet: Removed LNI 192.168.206.126@tcp39 [12864.816575] LNet: Removed LNI 192.168.206.126@tcp40 [12864.817685] LNet: Removed LNI 192.168.206.126@tcp41 [12864.818859] LNet: Removed LNI 192.168.206.126@tcp42 [12864.819893] LNet: Removed LNI 192.168.206.126@tcp43 [12864.821062] LNet: Removed LNI 192.168.206.126@tcp44 [12864.822339] LNet: Removed LNI 192.168.206.126@tcp45 [12864.823556] LNet: Removed LNI 192.168.206.126@tcp46 [12864.824713] LNet: Removed LNI 192.168.206.126@tcp47 [12864.825779] LNet: Removed LNI 192.168.206.126@tcp48 [12864.826910] LNet: Removed LNI 192.168.206.126@tcp49 [12864.834402] LNet: Removed LNI 192.168.206.126@tcp50 [12865.113308] Key type .llcrypt unregistered [12865.114564] Key type ._llcrypt unregistered [12871.426129] Key type ._llcrypt registered [12871.427359] Key type .llcrypt registered [12871.459928] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [12877.366889] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12877.644075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12877.655837] alg: No test for adler32 (adler32-zlib) [12878.497066] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [12878.577990] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [12880.160179] Key type lgssc registered [12880.527190] Lustre: Echo OBD driver; http://www.lustre.org/ [12891.181699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12891.188521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12892.266280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12892.276645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12892.310975] Lustre: lustre-MDT0000: new disk, initializing [12892.329451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12892.335894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12893.639962] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12897.627360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12897.670968] Lustre: 505629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12897.698161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12897.700218] Lustre: Skipped 1 previous similar message [12897.736334] Lustre: lustre-MDT0001: new disk, initializing [12897.774339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12897.787709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12897.793672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12899.115244] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12902.703955] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12904.483423] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12906.982411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12907.061875] Lustre: *** cfs_fail_loc=90f, val=0*** [12907.096486] Lustre: lustre-OST0000: new disk, initializing [12907.098340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12907.100957] Lustre: 507644:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12907.131729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12907.320367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12907.324667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12907.360970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12909.039940] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12912.497038] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12917.731204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12917.733560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.734844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12917.737869] Lustre: Skipped 1 previous similar message [12920.660481] Lustre: server umount lustre-OST0000 complete [12922.850662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12922.851975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12922.854320] Lustre: Skipped 1 previous similar message [12922.856021] Lustre: Skipped 1 previous similar message [12927.969437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12927.972449] Lustre: Skipped 2 previous similar messages [12928.096495] Lustre: server umount lustre-MDT0000 complete [12929.431257] LustreError: 505621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958420 with bad export cookie 4496771996433662071 [12929.435143] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12929.436550] LustreError: 505621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12929.563962] Lustre: server umount lustre-MDT0001 complete [12932.982766] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [12933.900154] Key type lgssc unregistered [12934.015343] LNet: 509546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12934.017946] LNetError: 509546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12934.026265] LNet: Removed LNI 192.168.206.126@tcp [12934.316789] Key type .llcrypt unregistered [12934.318145] Key type ._llcrypt unregistered [12941.287415] Key type ._llcrypt registered [12941.289180] Key type .llcrypt registered [12941.325074] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12941.702823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12941.710247] alg: No test for adler32 (adler32-zlib) [12942.566210] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [12942.656368] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [12944.240245] Key type lgssc registered [12944.687977] Lustre: Echo OBD driver; http://www.lustre.org/ [12947.377374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12947.386228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12948.562746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12950.037262] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12953.011668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12953.144329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12954.520817] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12956.902509] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12958.746478] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12961.133486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12961.273066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12963.397754] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12965.708397] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12979.608827] loop0: detected capacity change from 0 to 204800000 [12979.670593] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12979.672938] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12979.675557] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12981.521983] loop0: detected capacity change from 0 to 204800000 [12981.577504] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12981.580787] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12981.584088] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12983.273594] loop0: detected capacity change from 0 to 204800000 [12985.167413] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12985.215275] Lustre: fs15246-OST0000: new disk, initializing [12985.216813] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12985.218919] Lustre: 514416:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12985.236988] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12987.156655] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12991.041490] loop1: detected capacity change from 0 to 204800000 [12992.987770] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12993.031913] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12993.060896] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12993.105543] Lustre: fs15246-MDT0000: new disk, initializing [12993.168433] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12993.180598] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12993.233981] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12994.272862] Lustre: 515538:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006a264668 x1877956029873152/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790958500 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12994.637384] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [13003.360286] LustreError: 515538:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [13003.364797] LustreError: 515538:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [13003.370224] LustreError: 515538:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [13005.284723] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13005.289113] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13008.470932] Lustre: server umount fs15246-MDT0000 complete [13010.402462] LustreError: 510802:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.408284] LustreError: 510802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13011.780382] Lustre: server umount fs15246-OST0000 complete [13015.521609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13015.523324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13015.525432] Lustre: Skipped 1 previous similar message [13020.641631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13020.643399] Lustre: Skipped 2 previous similar messages [13025.761627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13025.764367] Lustre: Skipped 1 previous similar message [13029.344089] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13029.394807] Lustre: server umount lustre-OST0000 complete [13030.881097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13030.881509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13030.885931] Lustre: Skipped 1 previous similar message [13030.889230] Lustre: Skipped 1 previous similar message [13036.762570] Lustre: server umount lustre-MDT0000 complete [13038.043596] LustreError: 510789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958528 with bad export cookie 2715671513494365247 [13038.046969] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13038.047147] LustreError: 510789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13038.182648] Lustre: server umount lustre-MDT0001 complete [13041.737833] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [13042.794095] Key type lgssc unregistered [13042.909301] LNet: 518631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13042.911435] LNetError: 518631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13042.922239] LNet: Removed LNI 192.168.206.126@tcp [13043.223601] Key type .llcrypt unregistered [13043.225105] Key type ._llcrypt unregistered [13050.688228] Key type ._llcrypt registered [13050.689799] Key type .llcrypt registered [13050.728504] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [13051.183154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13051.202903] alg: No test for adler32 (adler32-zlib) [13052.075057] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [13052.179075] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [13053.760228] Key type lgssc registered [13054.200948] Lustre: Echo OBD driver; http://www.lustre.org/ [13066.348951] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [13069.218569] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [13080.026325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13080.035404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13081.115153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13081.124088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13081.156139] Lustre: lustre-MDT0000: new disk, initializing [13081.175515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13081.182544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13082.503181] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [13086.530978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13086.571500] Lustre: 523078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13086.593246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13086.595554] Lustre: Skipped 1 previous similar message [13086.631142] Lustre: lustre-MDT0001: new disk, initializing [13086.659590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13086.670089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13086.672976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13087.924596] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [13091.671124] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13093.611428] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13096.069982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13096.186395] Lustre: lustre-OST0000: new disk, initializing [13096.188463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13096.192346] Lustre: 525048:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13096.224580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13098.287704] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [13102.039384] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13103.480466] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13104.123130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13104.127591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13104.168121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13104.570168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [13105.934711] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13106.010590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13109.218474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13109.218804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13109.221229] LustreError: Skipped 1 previous similar message [13109.226094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13109.228419] Lustre: Skipped 1 previous similar message [13112.720955] Lustre: server umount lustre-OST0000 complete [13114.337581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13114.338329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13114.340871] Lustre: Skipped 2 previous similar messages [13114.343626] Lustre: Skipped 1 previous similar message [13119.457087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13119.459161] Lustre: Skipped 1 previous similar message [13120.091674] Lustre: server umount lustre-MDT0000 complete [13121.305393] LustreError: 523073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958611 with bad export cookie 5285828637247312236 [13121.308858] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13121.310171] LustreError: 523073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13121.427594] Lustre: server umount lustre-MDT0001 complete [13129.245264] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [13132.298463] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [13142.030908] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12831 sec ========= 12:30:31 (1790958631) [13142.536072] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:30:32 (1790958632) === [13143.586046] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:30:33 (1790958633) === [13156.077254] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [13157.083442] Key type lgssc unregistered [13157.207246] LNet: 531683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13157.209669] LNetError: 531683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13157.222366] LNet: Removed LNI 192.168.206.126@tcp [13157.523988] Key type .llcrypt unregistered [13157.525526] Key type ._llcrypt unregistered