[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489996524 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003164] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008260] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011007] pid_max: default: 32768 minimum: 301 [ 0.012136] LSM: Security Framework initializing [ 0.013054] Yama: becoming mindful. [ 0.014038] SELinux: Initializing. [ 0.015898] *** VALIDATE selinux *** [ 0.023267] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027841] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029018] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031137] *** VALIDATE tmpfs *** [ 0.033469] *** VALIDATE proc *** [ 0.035101] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.043840] debug: unmapping init [mem 0xffffffff9dc59000-0xffffffff9dc60fff] [ 0.045177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046776] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054307] rcu: Hierarchical SRCU implementation. [ 0.056524] smp: Bringing up secondary CPUs ... [ 0.057601] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.061336] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141280] node 0 deferred pages initialised in 75ms [ 0.145103] devtmpfs: initialized [ 0.146385] x86/mm: Memory block size: 128MB [ 0.149967] gcov: version magic: 0x41383552 [ 0.153311] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159307] pinctrl core: initialized pinctrl subsystem [ 0.161189] [ 0.161726] ************************************************************* [ 0.164013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166008] ** ** [ 0.168013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170012] ** ** [ 0.172013] ** This means that this kernel is built to expose internal ** [ 0.175013] ** IOMMU data structures, which may compromise security on ** [ 0.177011] ** your system. ** [ 0.179011] ** ** [ 0.182012] ** If you see this message and you are not debugging the ** [ 0.184011] ** kernel, report this immediately to your vendor! ** [ 0.186013] ** ** [ 0.188031] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191015] ************************************************************* [ 0.193681] NET: Registered protocol family 16 [ 0.195445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203116] cpuidle: using governor menu [ 0.204592] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205436] PCI: Using configuration type 1 for base access [ 0.206101] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220057] cryptd: max_cpu_qlen set to 1000 [ 0.224251] ACPI: Added _OSI(Module Device) [ 0.226013] ACPI: Added _OSI(Processor Device) [ 0.228012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.228970] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232239] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237416] ACPI: Interpreter enabled [ 0.239079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240012] ACPI: Using IOAPIC for interrupt routing [ 0.242130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246424] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256756] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269826] acpiphp: Slot [2] registered [ 0.270052] acpiphp: Slot [5] registered [ 0.270872] acpiphp: Slot [6] registered [ 0.272078] acpiphp: Slot [7] registered [ 0.273098] acpiphp: Slot [8] registered [ 0.274082] acpiphp: Slot [9] registered [ 0.274971] acpiphp: Slot [10] registered [ 0.276072] acpiphp: Slot [3] registered [ 0.276875] acpiphp: Slot [4] registered [ 0.278058] acpiphp: Slot [11] registered [ 0.278956] acpiphp: Slot [12] registered [ 0.280045] acpiphp: Slot [13] registered [ 0.281023] acpiphp: Slot [14] registered [ 0.281871] acpiphp: Slot [15] registered [ 0.282068] acpiphp: Slot [16] registered [ 0.283039] acpiphp: Slot [17] registered [ 0.283979] acpiphp: Slot [18] registered [ 0.285069] acpiphp: Slot [19] registered [ 0.286082] acpiphp: Slot [20] registered [ 0.287076] acpiphp: Slot [21] registered [ 0.288129] acpiphp: Slot [22] registered [ 0.290091] acpiphp: Slot [23] registered [ 0.291122] acpiphp: Slot [24] registered [ 0.292126] acpiphp: Slot [25] registered [ 0.293093] acpiphp: Slot [26] registered [ 0.294091] acpiphp: Slot [27] registered [ 0.296130] acpiphp: Slot [28] registered [ 0.297104] acpiphp: Slot [29] registered [ 0.299128] acpiphp: Slot [30] registered [ 0.300095] acpiphp: Slot [31] registered [ 0.301090] PCI host bridge to bus 0000:00 [ 0.303018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.305027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.314035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317020] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.319996] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.323350] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332733] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.337914] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.340016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348261] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349972] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354831] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.374015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.382783] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.390026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.424561] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.431058] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.438016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.455018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.464155] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.470013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.474016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.492022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.502000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.510015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.517026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.535021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.545746] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.556018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.563019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.577019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.594052] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.601014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.607018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.622019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.633680] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637067] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.640389] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.642360] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.645231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.649134] iommu: Default domain type: Passthrough [ 0.651480] SCSI subsystem initialized [ 0.653117] ACPI: bus type USB registered [ 0.655105] usbcore: registered new interface driver usbfs [ 0.657076] usbcore: registered new interface driver hub [ 0.658050] usbcore: registered new device driver usb [ 0.659116] pps_core: LinuxPPS API ver. 1 registered [ 0.660007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.662051] PTP clock support registered [ 0.663132] EDAC MC: Ver: 3.0.0 [ 0.665125] PCI: Using ACPI for IRQ routing [ 0.666880] NetLabel: Initializing [ 0.669011] NetLabel: domain hash size = 128 [ 0.670008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.672083] NetLabel: unlabeled traffic allowed by default [ 0.674141] vgaarb: loaded [ 0.675276] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.677015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.685973] clocksource: Switched to clocksource kvm-clock [ 0.793615] VFS: Disk quotas dquot_6.6.0 [ 0.794729] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.796269] *** VALIDATE ramfs *** [ 0.797041] *** VALIDATE hugetlbfs *** [ 0.797960] pnp: PnP ACPI init [ 0.799598] pnp: PnP ACPI: found 6 devices [ 0.814684] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.816923] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.818492] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.820652] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.823110] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.825490] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.828424] NET: Registered protocol family 2 [ 0.830965] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.835516] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.839110] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.843945] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.847465] TCP: Hash tables configured (established 65536 bind 65536) [ 0.850371] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.853304] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.855829] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.858515] NET: Registered protocol family 1 [ 0.861134] RPC: Registered named UNIX socket transport module. [ 0.863600] RPC: Registered udp transport module. [ 0.865436] RPC: Registered tcp transport module. [ 0.866838] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.869360] NET: Registered protocol family 44 [ 0.871184] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.873281] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.875243] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.877485] PCI: CLS 0 bytes, default 64 [ 0.879142] Unpacking initramfs... [ 2.280119] debug: unmapping init [mem 0xffff8e8ebcc54000-0xffff8e8ebffbffff] [ 2.283985] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.286303] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.288832] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.763310] Initialise system trusted keyrings [ 2.764561] Key type blacklist registered [ 2.766351] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.775304] zbud: loaded [ 2.778734] *** VALIDATE nfs *** [ 2.780170] *** VALIDATE nfs4 *** [ 2.781712] pstore: using deflate compression [ 2.785737] Platform Keyring initialized [ 2.888362] NET: Registered protocol family 38 [ 2.890275] Key type asymmetric registered [ 2.892310] Asymmetric key parser 'x509' registered [ 2.894139] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.897186] io scheduler mq-deadline registered [ 2.898909] io scheduler kyber registered [ 2.900590] io scheduler bfq registered [ 2.902516] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.905826] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.908552] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.911550] ACPI: Power Button [PWRF] [ 2.916287] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.921869] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.934773] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.942703] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.955293] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.981949] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.010323] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.014094] Non-volatile memory driver v1.3 [ 3.015120] Linux agpgart interface v0.103 [ 3.043186] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.045266] vda: detected capacity change from 0 to 77115392 [ 3.058686] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.060581] vdb: detected capacity change from 0 to 1073741824 [ 3.072523] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.075151] vdc: detected capacity change from 0 to 2621440000 [ 3.088437] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.091375] vdd: detected capacity change from 0 to 2621440000 [ 3.104914] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.107775] vde: detected capacity change from 0 to 4294967296 [ 3.122175] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.124696] vdf: detected capacity change from 0 to 4294967296 [ 3.130872] libphy: Fixed MDIO Bus: probed [ 3.135907] usbcore: registered new interface driver usbserial_generic [ 3.138285] usbserial: USB Serial support registered for generic [ 3.140078] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.142990] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.144821] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.147129] mousedev: PS/2 mouse device common for all mice [ 3.150126] rtc_cmos 00:05: RTC can wake from S4 [ 3.153085] rtc_cmos 00:05: registered as rtc0 [ 3.153178] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.154775] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.159478] intel_pstate: CPU model not supported [ 3.161684] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.162223] hid: raw HID events driver (C) Jiri Kosina [ 3.166352] usbcore: registered new interface driver usbhid [ 3.166484] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.167496] usbhid: USB HID core driver [ 3.171882] drop_monitor: Initializing network drop monitor service [ 3.173823] Initializing XFRM netlink socket [ 3.175101] NET: Registered protocol family 10 [ 3.177921] Segment Routing with IPv6 [ 3.178912] NET: Registered protocol family 17 [ 3.181025] mpls_gso: MPLS GSO support [ 3.187713] RAS: Correctable Errors collector initialized. [ 3.189492] AVX version of gcm_enc/dec engaged. [ 3.191109] AES CTR mode by8 optimization enabled [ 3.272610] sched_clock: Marking stable (3272585236, 0)->(4178208464, -905623228) [ 3.275656] registered taskstats version 1 [ 3.278648] Loading compiled-in X.509 certificates [ 3.280556] zswap: loaded using pool lzo/zbud [ 3.301730] Key type big_key registered [ 3.314760] Key type encrypted registered [ 3.316205] ima: No TPM chip found, activating TPM-bypass! [ 3.317980] ima: Allocated hash algorithm: sha1 [ 3.319540] ima: No architecture policies found [ 3.321105] evm: Initialising EVM extended attributes: [ 3.322928] evm: security.selinux [ 3.324077] evm: security.ima [ 3.325124] evm: security.capability [ 3.326425] evm: HMAC attrs: 0x1 [ 3.328633] rtc_cmos 00:05: setting system clock to 2026-10-02 20:03:27 UTC (1790971407) [ 3.334572] debug: unmapping init [mem 0xffffffff9ec03000-0xffffffff9edfffff] [ 3.337350] debug: unmapping init [mem 0xffffffff9d982000-0xffffffff9dc58fff] [ 3.346822] Write protecting the kernel read-only data: 28672k [ 3.349863] debug: unmapping init [mem 0xffffffff9c003000-0xffffffff9c1fffff] [ 3.351733] debug: unmapping init [mem 0xffffffff9c914000-0xffffffff9c9fffff] [ 3.379124] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.385636] systemd[1]: Detected virtualization kvm. [ 3.386729] systemd[1]: Detected architecture x86-64. [ 3.387783] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.414430] systemd[1]: No hostname configured. [ 3.415768] systemd[1]: Set hostname to . [ 3.417243] random: systemd: uninitialized urandom read (16 bytes read) [ 3.418938] systemd[1]: Initializing machine ID from random generator. [ 3.548177] random: systemd: uninitialized urandom read (16 bytes read) [ 3.550179] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.553659] random: systemd: uninitialized urandom read (16 bytes read) [ 3.555358] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.559464] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.105626] device-mapper: uevent: version 1.0.3 [ 4.107895] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.762975] virtio_net virtio0 ens2: renamed from eth0 [ 4.772409] random: fast init done [ 4.806321] scsi host0: ata_piix [ 4.833940] scsi host1: ata_piix [ 4.835431] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.837690] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.273346] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.841976] random: crng init done [ 9.843463] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.209822] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.391558] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.635305] SELinux: Disabled at runtime. [ 11.695384] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.703682] systemd[1]: Detected virtualization kvm. [ 11.705529] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.160398] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.163863] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.171824] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.176125] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.179739] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.186840] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.191271] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ 12.278972] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.668174] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.010358] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.042129] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.263457] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.273871] EDAC sbridge: Ver: 1.1.2 [ 14.933263] Key type dns_resolver registered [ 15.220780] NFS: Registering the id_resolver key type [ 15.222894] Key type id_resolver registered [ 15.224398] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-server login: [ 40.913493] libcfs: loading out-of-tree module taints kernel. [ 40.931672] Key type ._llcrypt registered [ 40.933016] Key type .llcrypt registered [ 40.989858] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 49.740286] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 50.444490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.450530] alg: No test for adler32 (adler32-zlib) [ 51.494453] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 51.833564] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 53.471291] Key type lgssc registered [ 54.531622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.153030] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 114.691585] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 128.504663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 128.554655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 129.885963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.936107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 130.044525] Lustre: lustre-MDT0000: new disk, initializing [ 130.103783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.134713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 130.725519] hrtimer: interrupt took 3770357 ns [ 134.340956] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.393329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.482580] Lustre: 6489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 147.519206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 147.523736] Lustre: Skipped 1 previous similar message [ 147.640682] Lustre: lustre-MDT0001: new disk, initializing [ 147.703820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 147.754279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 147.765739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 152.030537] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.103916] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.464127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 167.720759] Lustre: lustre-OST0000: new disk, initializing [ 167.727489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 167.734770] Lustre: 8427:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 167.822370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.125181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.138751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.204386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 174.187609] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.101410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.215559] Lustre: lustre-OST0001: new disk, initializing [ 190.219489] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 190.224512] Lustre: 9499:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.305576] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.136107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 197.145040] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 197.165207] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.219351] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 209.310201] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 217.423292] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 225.607765] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing check_logdir /tmp/testlogs/ [ 232.269891] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing yml_node [ 236.828238] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 239.419360] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 241.938478] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 243.676615] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 16:07:27 EDT 2026 [ 261.320875] Lustre: DEBUG MARKER: only running tests: 33-75 [ 262.957424] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 264.622571] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 284.129309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.130583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.153433] Lustre: Skipped 1 previous similar message [ 284.162604] Lustre: Skipped 3 previous similar messages [ 287.925719] Lustre: server umount lustre-MDT0000 complete [ 292.155220] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971696 with bad export cookie 5322884129546802730 [ 292.157096] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.169218] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 294.367621] LustreError: 6499:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.375838] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.383143] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.390876] LustreError: 6499:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 294.457676] Lustre: Skipped 3 previous similar messages [ 297.376367] LustreError: 10970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.377873] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.390514] LustreError: 10970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 297.409974] Lustre: Skipped 2 previous similar messages [ 298.628452] Lustre: server umount lustre-MDT0001 complete [ 303.020243] Lustre: server umount lustre-OST0000 complete [ 307.227928] Lustre: server umount lustre-OST0001 complete [ 314.156729] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 326.448618] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 367.350782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.622836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.650087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 367.854112] Lustre: lustre-MDT0000: new disk, initializing [ 367.916439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.931184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 371.626401] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 382.734834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.003419] Lustre: 15103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 383.065804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 383.068988] Lustre: Skipped 1 previous similar message [ 383.117338] Lustre: lustre-MDT0001: new disk, initializing [ 383.193223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 383.220839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 383.229986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 388.671963] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 399.877485] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 406.512038] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 412.216444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 412.510962] Lustre: lustre-OST0000: new disk, initializing [ 412.515567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 412.519817] Lustre: 17071:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 412.641842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.769662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 413.785605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 413.946685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 417.551281] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 427.139868] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 433.257544] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 433.482553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 438.747899] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 439.055919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 444.901586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.907262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 444.937114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 444.958838] Lustre: Skipped 1 previous similar message [ 450.026815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 450.039783] Lustre: Skipped 1 previous similar message [ 455.135146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 455.299266] Lustre: server umount lustre-OST0000 complete [ 460.256533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 460.259178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.271063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 460.288194] Lustre: Skipped 2 previous similar messages [ 460.310480] Lustre: Skipped 3 previous similar messages [ 465.584440] Lustre: server umount lustre-MDT0000 complete [ 468.471594] LustreError: 15096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971872 with bad export cookie 5322884129546804263 [ 468.473865] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 468.479476] LustreError: 15096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 469.079701] Lustre: server umount lustre-MDT0001 complete [ 472.801959] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:11:16 (1790971876) [ 475.026834] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 476.618974] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:11:20 (1790971880) [ 483.072617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.470500] LustreError: 19482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.527206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.936242] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 488.943267] LustreError: 19483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.028839] LustreError: 19482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.964217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.333193] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 509.061849] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 515.933372] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 523.630433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 530.158439] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 537.658669] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 547.812922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.826544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 547.828091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 547.842536] Lustre: Skipped 2 previous similar messages [ 553.808579] Lustre: server umount lustre-OST0000 complete [ 557.538986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 557.548414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.557336] Lustre: Skipped 1 previous similar message [ 563.575571] Lustre: server umount lustre-MDT0000 complete [ 567.437230] LustreError: 19462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971971 with bad export cookie 5322884129546805145 [ 567.458975] LustreError: 19462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 567.463399] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.717276] Lustre: server umount lustre-MDT0001 complete [ 580.902067] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 584.890673] Key type lgssc unregistered [ 585.358220] LNet: 23013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.387303] LNetError: 23013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.408858] LNet: Removed LNI 192.168.202.110@tcp [ 587.856155] Key type .llcrypt unregistered [ 587.857962] Key type ._llcrypt unregistered [ 611.993963] Key type ._llcrypt registered [ 611.997320] Key type .llcrypt registered [ 612.215305] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 613.402338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 613.606624] alg: No test for adler32 (adler32-zlib) [ 614.889821] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 615.178912] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 616.919931] Key type lgssc registered [ 618.456534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 626.904895] loop: module loaded [ 627.019254] loop0: detected capacity change from 0 to 204800000 [ 627.264544] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.272102] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.465502] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.339106] loop0: detected capacity change from 0 to 204800000 [ 633.611708] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.627708] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.646685] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.845727] loop0: detected capacity change from 0 to 204800000 [ 645.312688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 645.344530] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.666801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 646.797001] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 646.919255] Lustre: lustre-MDT0000: new disk, initializing [ 647.045815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.071529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 651.860605] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 662.398507] loop1: detected capacity change from 0 to 204800000 [ 669.361718] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.790469] Lustre: lustre-OST07c6: new disk, initializing [ 669.796496] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 669.802728] Lustre: Skipped 1 previous similar message [ 669.819543] Lustre: 26214:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 669.933060] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 675.980797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 675.991737] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 676.034828] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 676.798506] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 681.314701] Lustre: 26498:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 681.326843] Lustre: 26498:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 681.333433] Lustre: 26498:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 681.346062] Lustre: 26498:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 681.351447] Lustre: 26498:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 681.357094] Lustre: 26498:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 693.931528] Lustre: 25200:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 693.941153] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 693.947164] Lustre: 25200:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.966591] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 693.985730] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 694.001407] Lustre: 25200:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.903244] Lustre: 25200:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 698.912921] Lustre: 25200:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 698.918689] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 698.925104] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.931267] Lustre: 25200:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.937506] Lustre: 25200:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.943594] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 698.953032] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.960166] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 698.966290] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.975933] Lustre: 25200:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.985344] Lustre: 25200:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.164879] Lustre: 25201:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 702.180137] Lustre: 25201:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 702.190455] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 702.209031] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.222246] Lustre: 25201:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 702.236226] Lustre: 25201:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.257229] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 702.266048] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.281646] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 702.287187] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.292424] Lustre: 25201:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 702.300497] Lustre: 25201:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.041807] Lustre: 25202:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 707.054596] Lustre: 25202:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 707.066264] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 707.083883] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.094103] Lustre: 25202:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.102138] Lustre: 25202:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.109554] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 707.116879] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.122133] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.132165] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.139975] Lustre: 25202:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.153090] Lustre: 25202:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.451141] Lustre: 25200:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 718.478961] Lustre: 25200:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 718.485959] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 718.499663] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.509551] Lustre: 25200:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.523338] Lustre: 25200:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.533707] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 718.547120] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.557476] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 718.568398] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.575403] Lustre: 25200:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 718.582569] Lustre: 25200:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 767.971639] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.979505] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 771.699234] Lustre: server umount lustre-OST07c6 complete [ 783.377275] Lustre: server umount lustre-MDT0000 complete [ 801.270550] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 804.375710] Key type lgssc unregistered [ 804.702626] LNet: 28768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.715302] LNetError: 28768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.728552] LNet: Removed LNI 192.168.202.110@tcp [ 805.829096] Key type .llcrypt unregistered [ 805.832117] Key type ._llcrypt unregistered [ 827.665477] Key type ._llcrypt registered [ 827.667474] Key type .llcrypt registered [ 827.764065] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 828.978276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 829.015265] alg: No test for adler32 (adler32-zlib) [ 830.011637] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 830.217779] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 831.855262] Key type lgssc registered [ 833.171337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 840.908961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 840.955103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.374410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.945328] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 851.939839] LustreError: 30054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 851.956327] LustreError: 30054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 856.412155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.775141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 861.697227] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 869.596669] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.225847] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 881.128852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 881.434067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.575132] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 887.611161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 895.830172] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 913.208608] Lustre: server umount lustre-OST0000 complete [ 918.496450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 918.499475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.504670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 918.535893] Lustre: Skipped 1 previous similar message [ 923.617717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.623500] Lustre: Skipped 2 previous similar messages [ 928.739198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.753182] Lustre: Skipped 1 previous similar message [ 929.762093] Lustre: 29356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790972317/real 1790972317] req@00000000b2bb12e5 x1877970501267712/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790972333 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 929.793413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.295134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 931.571544] Lustre: server umount lustre-MDT0000 complete [ 933.920054] Lustre: 29356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790972322/real 1790972322] req@000000002f652f61 x1877970501270144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790972338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 934.881153] LustreError: 31445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.895727] LustreError: 31445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 935.290833] LustreError: 30035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972339 with bad export cookie 9847511023578628140 [ 935.299048] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 935.662237] Lustre: server umount lustre-MDT0001 complete [ 947.738507] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 951.285301] Key type lgssc unregistered [ 951.604031] LNet: 33791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.612689] LNetError: 33791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.631183] LNet: Removed LNI 192.168.202.110@tcp [ 952.803870] Key type .llcrypt unregistered [ 952.809213] Key type ._llcrypt unregistered [ 972.222611] Key type ._llcrypt registered [ 972.226459] Key type .llcrypt registered [ 972.381380] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 973.955462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 974.140073] alg: No test for adler32 (adler32-zlib) [ 975.308281] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 975.594934] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 977.319267] Key type lgssc registered [ 978.600064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.021321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 986.036829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.689617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.986861] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 996.834856] LustreError: 35079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.856229] LustreError: 35079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1000.559942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.031639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1006.028056] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1013.171797] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1018.719086] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1024.733764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1025.016070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1030.257787] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1032.195739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1036.775630] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1046.499574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1046.510238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.524732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.522893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.530110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1047.546228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.807745] Lustre: server umount lustre-OST0000 complete [ 1057.250601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1057.255315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.267370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.271649] Lustre: Skipped 1 previous similar message [ 1061.521421] Lustre: server umount lustre-MDT0000 complete [ 1062.881620] LustreError: 36920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.902249] LustreError: 36920:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1064.565202] LustreError: 35058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972468 with bad export cookie 9422780102719358515 [ 1064.580723] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.977620] Lustre: server umount lustre-MDT0001 complete [ 1074.681602] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1077.590673] Key type lgssc unregistered [ 1077.961231] LNet: 38610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.972237] LNetError: 38610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.989041] LNet: Removed LNI 192.168.202.110@tcp [ 1078.907285] Key type .llcrypt unregistered [ 1078.909710] Key type ._llcrypt unregistered [ 1101.231796] Key type ._llcrypt registered [ 1101.234390] Key type .llcrypt registered [ 1101.351934] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1102.314236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.449481] alg: No test for adler32 (adler32-zlib) [ 1103.551751] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1103.772545] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1105.439284] Key type lgssc registered [ 1106.552301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1113.827878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1113.844295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.449130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1119.780805] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1124.836205] LustreError: 40035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.863151] LustreError: 40035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1128.845801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.283581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1135.330755] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1145.536399] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1152.225823] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1160.356747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1160.867615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.479323] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1174.982327] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1185.509642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1196.007917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.010284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1196.017359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.021553] Lustre: Skipped 2 previous similar messages [ 1197.099427] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1197.108370] Lustre: Skipped 2 previous similar messages [ 1198.418282] Lustre: server umount lustre-MDT0000 complete [ 1201.125656] LustreError: 40030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.143974] LustreError: 40030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1202.194423] LustreError: 40029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.308483] LustreError: 40016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972606 with bad export cookie 3028792173666360695 [ 1202.317615] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.240988] LustreError: 40030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.241565] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.292934] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1208.807873] Lustre: server umount lustre-MDT0001 complete [ 1230.931832] Lustre: server umount lustre-OST0000 complete [ 1249.192714] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1252.290558] Key type lgssc unregistered [ 1252.581875] LNet: 43917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.588606] LNetError: 43917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.606432] LNet: Removed LNI 192.168.202.110@tcp [ 1254.754981] Key type .llcrypt unregistered [ 1254.764136] Key type ._llcrypt unregistered [ 1278.777388] Key type ._llcrypt registered [ 1278.781085] Key type .llcrypt registered [ 1278.854791] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1280.145874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.161814] alg: No test for adler32 (adler32-zlib) [ 1281.358488] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1281.742799] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1283.423220] Key type lgssc registered [ 1284.825267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1294.193824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1294.208939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.606750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1300.841170] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1305.066271] LustreError: 45350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1305.088097] LustreError: 45350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1310.177230] LustreError: 45351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.392909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.824018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1315.503524] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1323.684221] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1328.759197] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1334.581790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1334.933267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.137356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1341.038575] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1349.309106] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1355.744554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1355.751805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.762841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.781476] Lustre: Skipped 1 previous similar message [ 1360.869433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1360.878249] Lustre: Skipped 3 previous similar messages [ 1361.256741] Lustre: server umount lustre-OST0000 complete [ 1381.344678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.347474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1381.362735] Lustre: Skipped 1 previous similar message [ 1381.375175] Lustre: Skipped 2 previous similar messages [ 1386.049977] Lustre: server umount lustre-MDT0000 complete [ 1386.465925] LustreError: 47256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1386.478424] LustreError: 47256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1390.190740] LustreError: 45332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972794 with bad export cookie 17479296830157374039 [ 1390.197410] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1390.204438] LustreError: 45332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1396.646448] Lustre: server umount lustre-MDT0001 complete [ 1408.506905] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1412.375782] Key type lgssc unregistered [ 1412.764711] LNet: 49061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.773457] LNetError: 49061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.794974] LNet: Removed LNI 192.168.202.110@tcp [ 1414.532579] Key type .llcrypt unregistered [ 1414.538434] Key type ._llcrypt unregistered [ 1442.948221] Key type ._llcrypt registered [ 1442.949723] Key type .llcrypt registered [ 1443.091840] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1444.656767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1444.724916] alg: No test for adler32 (adler32-zlib) [ 1446.027946] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1446.346723] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1448.112570] Key type lgssc registered [ 1449.161404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1457.845779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1457.864486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.395375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.045510] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1468.897322] LustreError: 50493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1468.909880] LustreError: 50493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1472.322907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.808873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.867684] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1486.011071] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1492.653400] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1499.892953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1500.350230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.464120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1507.650533] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1515.656801] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1522.670935] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1526.477155] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1538.479709] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-10-0216h29m01s [ 1540.840595] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1544.235746] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1544.684769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1544.698909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.755300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.757528] Lustre: Skipped 1 previous similar message [ 1545.835995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1545.848894] Lustre: Skipped 3 previous similar messages [ 1550.351229] LustreError: 51657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.555578] Lustre: server umount lustre-MDT0000 complete [ 1552.389854] LustreError: 50494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.420621] LustreError: 50494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1555.561894] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1556.497054] LustreError: 50489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.645266] LustreError: 51657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.667157] LustreError: 51657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1563.170107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.309063] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1563.566855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.658799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1564.594263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1568.790304] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1572.620505] Lustre: DEBUG MARKER: Wait for df (36887) ... [ 1574.367635] Lustre: DEBUG MARKER: done [ 1578.983832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1578.990672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.000252] Lustre: Skipped 2 previous similar messages [ 1579.006720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1584.575751] Lustre: server umount lustre-OST0000 complete [ 1590.241131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1590.244841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.247194] LustreError: Skipped 1 previous similar message [ 1590.248423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.262325] Lustre: Skipped 2 previous similar messages [ 1590.278847] Lustre: Skipped 4 previous similar messages [ 1595.206784] Lustre: server umount lustre-MDT0000 complete [ 1597.424228] LustreError: 50494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.441178] LustreError: 50494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1599.033261] LustreError: 51228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973003 with bad export cookie 7532934048655184684 [ 1599.043244] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1599.048935] LustreError: 51228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1599.419677] Lustre: server umount lustre-MDT0001 complete [ 1611.589468] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1613.891263] Key type lgssc unregistered [ 1614.127738] LNet: 55418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.141280] LNetError: 55418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.155393] LNet: Removed LNI 192.168.202.110@tcp [ 1614.930525] Key type .llcrypt unregistered [ 1614.942049] Key type ._llcrypt unregistered [ 1657.820690] Key type ._llcrypt registered [ 1657.825924] Key type .llcrypt registered [ 1657.932589] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1659.361776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1659.396960] alg: No test for adler32 (adler32-zlib) [ 1660.452074] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1660.695858] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1662.399234] Key type lgssc registered [ 1663.594161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1670.539834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1670.552426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.802881] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1671.820227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1671.992774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.376937] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1685.307883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.367821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1685.393832] Lustre: 57641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1685.653839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1690.337466] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1696.921682] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1702.680517] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1710.273957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1710.469746] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1710.629878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.902408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1717.360927] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1727.204433] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1733.324386] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-10-0216h32m17s [ 1734.876738] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1742.222966] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1745.451462] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1745.515458] LustreError: 58667:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1765.954099] Lustre: lustre-MDT0000: Client 7a172525-0197-4dcf-8a52-e373c9ba4514 (at 192.168.202.10@tcp) reconnecting [ 1785.527494] LustreError: 58667:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1787.946592] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1790.953848] Lustre: DEBUG MARKER: done [ 1802.722488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1802.728874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.740051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.233559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.247888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.250699] Lustre: Skipped 1 previous similar message [ 1804.718701] Lustre: server umount lustre-OST0000 complete [ 1808.359276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.391388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.409140] Lustre: Skipped 1 previous similar message [ 1814.498340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1814.513594] Lustre: Skipped 3 previous similar messages [ 1814.700802] Lustre: server umount lustre-MDT0000 complete [ 1818.771048] LustreError: 60402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973222 with bad export cookie 12320601103567228943 [ 1818.776832] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.783604] LustreError: 60402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1819.169787] Lustre: server umount lustre-MDT0001 complete [ 1831.164615] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1834.566856] Key type lgssc unregistered [ 1834.934906] LNet: 61971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.946189] LNetError: 61971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.986985] LNet: Removed LNI 192.168.202.110@tcp [ 1836.001524] Key type .llcrypt unregistered [ 1836.004070] Key type ._llcrypt unregistered [ 1884.728570] Key type ._llcrypt registered [ 1884.730576] Key type .llcrypt registered [ 1884.950959] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1886.064364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.333691] alg: No test for adler32 (adler32-zlib) [ 1887.407421] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1887.684839] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1889.391300] Key type lgssc registered [ 1890.857221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.729521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1894.745387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.246157] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1896.303500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1896.583652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.739963] Lustre: Failing over lustre-MDT0000 [ 1898.063475] Lustre: server umount lustre-MDT0000 complete [ 1905.500523] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:35:09 (1790973309) [ 1912.448263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.943353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.688128] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1926.161746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.240418] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1926.283915] Lustre: 64788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1926.574587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.124824] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1939.184681] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1945.108641] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1951.758519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1951.946720] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1952.218590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.140798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1959.212978] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1966.887326] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1973.274207] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1979.876177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.880775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.900029] Lustre: Skipped 1 previous similar message [ 1979.915513] Lustre: Skipped 2 previous similar messages [ 1984.998480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1985.004304] Lustre: Skipped 2 previous similar messages [ 1988.020877] Lustre: server umount lustre-MDT0000 complete [ 1991.453574] LustreError: 64781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973395 with bad export cookie 8579556381291742155 [ 1991.456077] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1991.463016] LustreError: 64781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1991.831862] Lustre: server umount lustre-MDT0001 complete [ 1994.808620] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1998.815385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2006.304668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.615557] Lustre: 63699:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790973399/real 1790973399] req@00000000fbe7c3d0 x1877971609625344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790973415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2011.668871] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.691287] Lustre: Skipped 1 previous similar message [ 2012.639756] Lustre: 63700:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790973400/real 1790973400] req@000000004b08d215 x1877971609625472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790973416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2016.735706] Lustre: 63702:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790973404/real 1790973404] req@00000000e95e8ef2 x1877971609625600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790973420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2016.808419] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7710b51285548cb5 [ 2016.815195] Lustre: MGC192.168.202.110@tcp: Connection restored to 0@lo (at 0@lo) [ 2017.036446] LustreError: 66638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.048539] LustreError: 66638:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2017.107872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.208445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2020.195745] LustreError: 66638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.197047] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2020.590866] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2021.231426] LustreError: 68558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.251917] LustreError: 68558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2022.239318] Lustre: 63700:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790973410/real 1790973410] req@000000007b585445 x1877971609625984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790973426 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2022.251960] Lustre: 63700:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2026.472371] LustreError: 66638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.483668] LustreError: 66638:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2027.276492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.488158] Lustre: 63700:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790973415/real 1790973415] req@000000009f15d312 x1877971609626112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790973431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2027.669578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2031.578427] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2033.145363] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2038.380677] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2043.335923] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2051.406968] Lustre: server umount lustre-MDT0000 complete [ 2053.091742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2053.098875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.123197] LustreError: 66638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.836086] LustreError: 64781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973459 with bad export cookie 8579556381291744437 [ 2054.847836] LustreError: 64781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2054.848694] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.155881] Lustre: server umount lustre-MDT0001 complete [ 2060.673123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2069.263696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.079482] Lustre: 63699:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790973462/real 1790973462] req@00000000fc560e3b x1877971609651200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790973478 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2074.103170] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.114511] Lustre: Skipped 2 previous similar messages [ 2080.304845] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7710b51285549218 [ 2080.333983] Lustre: MGC192.168.202.110@tcp: Connection restored to 0@lo (at 0@lo) [ 2080.682174] LustreError: 66639:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.701633] LustreError: 66639:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2080.784232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.848849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2085.393731] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2093.043761] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2093.831843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.406382] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2106.778811] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2112.694079] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2119.654453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2119.655907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.671474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.679775] Lustre: Skipped 1 previous similar message [ 2124.770807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.783982] Lustre: Skipped 2 previous similar messages [ 2125.948905] Lustre: server umount lustre-MDT0000 complete [ 2129.458385] LustreError: 64782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973533 with bad export cookie 8579556381291745816 [ 2129.460987] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.467669] LustreError: 64782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2129.891890] LustreError: 72267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.892236] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.916825] LustreError: 72267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2129.950419] Lustre: Skipped 2 previous similar messages [ 2130.135529] Lustre: server umount lustre-MDT0001 complete [ 2136.357734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2139.700224] Lustre: DEBUG MARKER: files compared the same [ 2141.642831] Lustre: server umount lustre-OST0000 complete [ 2156.643241] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2159.075379] Key type lgssc unregistered [ 2159.416624] LNet: 74516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2159.420975] LNetError: 74516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2159.433460] LNet: Removed LNI 192.168.202.110@tcp [ 2160.442705] Key type .llcrypt unregistered [ 2160.446245] Key type ._llcrypt unregistered [ 2185.337577] Key type ._llcrypt registered [ 2185.340086] Key type .llcrypt registered [ 2185.447598] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2186.648886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2186.688238] alg: No test for adler32 (adler32-zlib) [ 2187.695434] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2187.905929] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2189.567218] Key type lgssc registered [ 2190.751139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2196.856575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2196.878891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.243099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.254840] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug +malloc all [ 2208.739945] LustreError: 75951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.761714] LustreError: 75951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2212.986835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.511731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.407077] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug +malloc all [ 2229.683511] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2235.387447] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2242.434330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2242.703386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.758570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2247.965572] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug +malloc all [ 2255.795957] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2264.546119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.553492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.571216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2268.640506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.640545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2268.641924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2270.369250] Lustre: server umount lustre-OST0000 complete [ 2278.884911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.906771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.913343] Lustre: Skipped 2 previous similar messages [ 2281.666799] Lustre: server umount lustre-MDT0000 complete [ 2284.008278] LustreError: 75947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.038942] LustreError: 75947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2286.161318] LustreError: 75931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973690 with bad export cookie 6605091301088911562 [ 2286.168689] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.172811] LustreError: 75931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2286.776883] Lustre: server umount lustre-MDT0001 complete [ 2299.742972] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2302.351455] Key type lgssc unregistered [ 2302.633402] LNet: 79481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.636953] LNetError: 79481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.653147] LNet: Removed LNI 192.168.202.110@tcp [ 2304.626576] Key type .llcrypt unregistered [ 2304.628930] Key type ._llcrypt unregistered [ 2328.279495] Key type ._llcrypt registered [ 2328.281190] Key type .llcrypt registered [ 2328.411718] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2329.621443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2329.958019] alg: No test for adler32 (adler32-zlib) [ 2331.295534] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2331.764655] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2333.585328] Key type lgssc registered [ 2335.099404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2345.255396] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2345.281878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2361.055842] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2366.176195] LustreError: 80889:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.110@tcp: failed processing log, type 4: rc = -110 [ 2392.800268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.674255] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2406.304682] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2413.321670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.483873] Lustre: *** cfs_fail_loc=706, val=0*** [ 2413.753781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.835362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2418.520072] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2419.175434] LustreError: 80913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.190817] LustreError: 80913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2424.291470] LustreError: 80914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.373687] LustreError: 80913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.420436] LustreError: 80913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2428.083580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.363051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2432.448404] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2440.114725] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2448.098962] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2453.987332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2453.995691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2454.013288] Lustre: Skipped 1 previous similar message [ 2457.865684] Lustre: server umount lustre-OST0000 complete [ 2464.230861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.237128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2464.250744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.257384] Lustre: Skipped 2 previous similar messages [ 2469.277155] Lustre: server umount lustre-MDT0000 complete [ 2469.363307] LustreError: 81977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.418040] LustreError: 81977:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2474.480580] LustreError: 81977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.518792] LustreError: 81977:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2474.819790] LustreError: 80895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973878 with bad export cookie 2095925779707688532 [ 2474.829779] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2474.834027] LustreError: 80895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2475.152686] Lustre: server umount lustre-MDT0001 complete [ 2487.816877] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2491.444647] Key type lgssc unregistered [ 2491.789286] LNet: 84503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.803590] LNetError: 84503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2491.842088] LNet: Removed LNI 192.168.202.110@tcp [ 2493.202282] Key type .llcrypt unregistered [ 2493.208173] Key type ._llcrypt unregistered [ 2521.359522] Key type ._llcrypt registered [ 2521.364380] Key type .llcrypt registered [ 2521.481116] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2523.128346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2523.167074] alg: No test for adler32 (adler32-zlib) [ 2524.174548] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2524.370198] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2526.047306] Key type lgssc registered [ 2527.613311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2535.690255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2535.715082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.641779] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2549.169674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.523714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.708837] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2554.867395] LustreError: 86624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.885305] LustreError: 86624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2558.948326] LustreError: 86628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.964973] LustreError: 86628:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2562.359898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.855242] LustreError: 86624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.895239] LustreError: 86624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2562.914791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.162660] LustreError: 86627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.184195] LustreError: 86627:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2569.440769] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2573.300390] LustreError: 86622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.335084] LustreError: 86622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2577.860589] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.878323] Lustre: MGS: Client a4ac64e0-ed69-42d3-a060-25e987464140 (at 0@lo) reconnecting [ 2577.915351] LustreError: 86638:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2578.171250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.265936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2584.258643] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2604.009358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.022560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.041848] Lustre: Skipped 1 previous similar message [ 2604.046381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2606.578249] Lustre: server umount lustre-OST0000 complete [ 2614.240489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2614.243378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.258410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.261631] Lustre: Skipped 1 previous similar message [ 2614.284993] Lustre: Skipped 2 previous similar messages [ 2616.393523] Lustre: server umount lustre-MDT0000 complete [ 2619.361364] LustreError: 86628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.399144] LustreError: 86628:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2622.313647] Lustre: server umount lustre-MDT0001 complete [ 2626.794907] Lustre: server umount MGS complete [ 2639.381983] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:47:23 (1790974043) [ 2665.912357] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 2676.855714] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2715.751800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.522893] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2730.205035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.467847] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2730.556508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2730.686206] Lustre: lustre-MDT0001: new disk, initializing [ 2730.760514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2730.769801] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2730.776251] Lustre: 93814:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2730.781141] LustreError: 93814:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2734.643968] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2745.312295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2745.852568] Lustre: lustre-OST0000: new disk, initializing [ 2745.861438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2745.873304] Lustre: 94582:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2746.027108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.696549] Lustre: 94864:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000623f509c x1877972277568768/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790974173 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2755.089854] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2767.446716] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2773.985191] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2777.040267] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2777.054581] Lustre: MGS: Client 2a031cc1-ed26-479f-b7ac-50c5bf75077b (at 0@lo) reconnecting [ 2777.097987] Lustre: 93081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2777.120598] LustreError: 93800:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2777.136208] LustreError: 93800:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2777.147479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2777.226601] Lustre: lustre-MDT0000: new disk, initializing [ 2777.377688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.405750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2778.469694] Lustre: 95651:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000affbfb9d x1877972277576320/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790974198 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2783.204890] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2783.519861] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2787.573185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2787.595249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2787.732123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2811.880501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.894642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2811.897311] Lustre: Skipped 1 previous similar message [ 2816.694336] Lustre: server umount lustre-OST0000 complete [ 2822.112898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.113700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2822.125703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.129223] Lustre: Skipped 2 previous similar messages [ 2827.233116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.238772] Lustre: Skipped 1 previous similar message [ 2834.399129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2834.485238] Lustre: server umount lustre-MDT0000 complete [ 2837.474791] LustreError: 96078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.487746] LustreError: 96078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2845.650665] Lustre: server umount lustre-MDT0001 complete [ 2850.067550] Lustre: server umount MGS complete [ 2866.405917] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:51:09 (1790974269) [ 2891.111586] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2894.521993] Key type lgssc unregistered [ 2895.065821] LNet: 98152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2895.080392] LNetError: 98152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2895.106899] LNet: Removed LNI 192.168.202.110@tcp [ 2896.255669] Key type .llcrypt unregistered [ 2896.257038] Key type ._llcrypt unregistered [ 2914.274966] Key type ._llcrypt registered [ 2914.280890] Key type .llcrypt registered [ 2914.405160] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2916.093280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2916.196767] alg: No test for adler32 (adler32-zlib) [ 2917.361151] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2917.644838] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2919.415252] Key type lgssc registered [ 2921.013670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2931.494954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2931.550761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.435670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.975549] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2942.950857] LustreError: 99388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.987981] LustreError: 99388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2948.064572] LustreError: 99389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.815606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2949.126140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2954.275101] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2963.316385] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2970.176917] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2977.057853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2977.591311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.619526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2985.074457] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2993.688382] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3004.390060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3004.393294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.400757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3004.407282] LustreError: Skipped 1 previous similar message [ 3004.438812] Lustre: Skipped 1 previous similar message [ 3009.253884] Lustre: server umount lustre-OST0000 complete [ 3014.627710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.646403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3014.649019] Lustre: Skipped 1 previous similar message [ 3019.747837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.765425] Lustre: Skipped 1 previous similar message [ 3020.236426] Lustre: server umount lustre-MDT0000 complete [ 3023.852029] LustreError: 99383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.874844] LustreError: 99383:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3024.921948] LustreError: 99370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974429 with bad export cookie 4564614844192633397 [ 3024.933859] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3024.943101] LustreError: 99370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3025.353879] Lustre: server umount lustre-MDT0001 complete [ 3037.662094] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3043.562881] Key type lgssc unregistered [ 3044.017771] LNet: 102923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3044.024465] LNetError: 102923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3044.038978] LNet: Removed LNI 192.168.202.110@tcp [ 3045.337749] Key type .llcrypt unregistered [ 3045.339818] Key type ._llcrypt unregistered [ 3064.394614] Key type ._llcrypt registered [ 3064.399494] Key type .llcrypt registered [ 3064.645931] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3066.151188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3066.171813] alg: No test for adler32 (adler32-zlib) [ 3067.119674] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3067.315266] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3068.937038] Key type lgssc registered [ 3069.996800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3076.390478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3076.402381] LustreError: 104018:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3076.442401] LustreError: 104022:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3076.457740] LustreError: 104018:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4954 [ 3076.469462] LustreError: 104018:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3076.495689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.497287] LustreError: 104018:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3078.200488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.534979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.964987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3093.743606] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3100.853116] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing lsmod [ 3104.805656] LustreError: 105222:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3104.876211] LustreError: 105229:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3104.888199] LustreError: 105222:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4925 [ 3104.893517] LustreError: 105222:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3104.906269] LustreError: 105222:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3104.915121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.445573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.882869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.885127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3110.911387] Lustre: Skipped 1 previous similar message [ 3110.929578] Lustre: Skipped 2 previous similar messages [ 3112.999581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3113.017789] Lustre: Skipped 2 previous similar messages [ 3118.050953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.057642] Lustre: Skipped 1 previous similar message [ 3123.167136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3123.169406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.183719] Lustre: Skipped 3 previous similar messages [ 3123.361276] Lustre: server umount lustre-MDT0000 complete [ 3127.342185] LustreError: 104035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974531 with bad export cookie 10542486524130966269 [ 3127.348577] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3127.357241] LustreError: 104035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3127.392236] LustreError: 104490:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005a78475c x1877972846926592/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3127.728750] Lustre: server umount lustre-MDT0001 complete [ 3141.330315] Lustre: server umount lustre-OST0000 complete [ 3151.759710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3152.449363] LustreError: 106296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.472478] LustreError: 106296:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3152.569846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.802345] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3157.994629] LustreError: 106297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.080742] LustreError: 106296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.212042] LustreError: 106297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.706927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.399839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3174.055436] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3182.509737] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3188.473931] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3196.549305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.989718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.024475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3204.318612] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3212.994396] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3223.534245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.537354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3223.540705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3223.552097] Lustre: Skipped 2 previous similar messages [ 3226.448222] Lustre: server umount lustre-OST0000 complete [ 3233.761426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3233.782463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3233.787879] Lustre: Skipped 1 previous similar message [ 3233.817866] Lustre: Skipped 2 previous similar messages [ 3236.741768] Lustre: server umount lustre-MDT0000 complete [ 3238.894575] LustreError: 106292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.929463] LustreError: 106292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3241.238894] LustreError: 106277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974645 with bad export cookie 10542486524130966976 [ 3241.241467] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.254992] LustreError: 106277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3241.725586] Lustre: server umount lustre-MDT0001 complete [ 3254.329780] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3257.644744] Key type lgssc unregistered [ 3257.961512] LNet: 109829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3257.966444] LNetError: 109829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3257.976226] LNet: Removed LNI 192.168.202.110@tcp [ 3258.814279] Key type .llcrypt unregistered [ 3258.818416] Key type ._llcrypt unregistered [ 3286.901331] Key type ._llcrypt registered [ 3286.903965] Key type .llcrypt registered [ 3287.071322] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3290.141370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3290.210159] alg: No test for adler32 (adler32-zlib) [ 3291.629480] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3291.796940] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3293.497720] Key type lgssc registered [ 3296.153533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3308.667909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3308.695821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.699419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.395101] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3320.803555] LustreError: 111265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.831423] LustreError: 111265:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3325.923871] LustreError: 111264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3326.117480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.514256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3332.464842] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3340.797196] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3346.259312] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3354.047139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3354.346336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.397189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3361.648478] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3371.868940] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3378.138127] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3382.369819] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3386.341076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3386.353700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.380392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3387.362108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3387.373218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3387.383950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3392.486640] LustreError: 113086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.583536] Lustre: server umount lustre-OST0000 complete [ 3397.604712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.624862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.628561] Lustre: Skipped 1 previous similar message [ 3402.723799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.557109] Lustre: server umount lustre-MDT0000 complete [ 3407.757146] LustreError: 111245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974811 with bad export cookie 532248542119178610 [ 3407.758831] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3407.773849] LustreError: 111245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3408.267510] Lustre: server umount lustre-MDT0001 complete [ 3421.597938] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3424.985881] Key type lgssc unregistered [ 3425.282169] LNet: 114902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3425.289559] LNetError: 114902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3426.342243] LNet: Removed LNI 192.168.202.110@tcp [ 3427.397652] Key type .llcrypt unregistered [ 3427.405275] Key type ._llcrypt unregistered [ 3448.439668] Key type ._llcrypt registered [ 3448.442346] Key type .llcrypt registered [ 3448.605807] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3449.936115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3449.978068] alg: No test for adler32 (adler32-zlib) [ 3451.294236] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3451.570562] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3453.271402] Key type lgssc registered [ 3454.740638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3463.834143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3463.849419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.314466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.222149] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3475.427724] LustreError: 116154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.461805] LustreError: 116154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3479.058201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.489995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3484.211528] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3492.240908] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3498.604920] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3504.894751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3505.219856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.311526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3511.375994] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3521.169838] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3534.820248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3534.826413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3534.831592] Lustre: Skipped 1 previous similar message [ 3534.832490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3534.835363] Lustre: Skipped 1 previous similar message [ 3539.940409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3539.957785] Lustre: Skipped 1 previous similar message [ 3541.140697] Lustre: server umount lustre-OST0000 complete [ 3545.059055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.063929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3545.065801] Lustre: Skipped 1 previous similar message [ 3545.071514] Lustre: Skipped 1 previous similar message [ 3550.176040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3555.299408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3555.312294] Lustre: Skipped 2 previous similar messages [ 3559.394243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3559.652534] Lustre: server umount lustre-MDT0000 complete [ 3560.417050] LustreError: 118509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.455194] LustreError: 118509:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3564.006053] LustreError: 116134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974968 with bad export cookie 11936528757882310804 [ 3564.014262] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3564.019086] LustreError: 116134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3564.332137] Lustre: server umount lustre-MDT0001 complete [ 3576.827458] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3580.117915] Key type lgssc unregistered [ 3580.499524] LNet: 119713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3580.508880] LNetError: 119713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3581.542158] LNet: Removed LNI 192.168.202.110@tcp [ 3582.814744] Key type .llcrypt unregistered [ 3582.816663] Key type ._llcrypt unregistered [ 3610.692244] Key type ._llcrypt registered [ 3610.694320] Key type .llcrypt registered [ 3610.867606] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3612.519805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3612.581957] alg: No test for adler32 (adler32-zlib) [ 3613.905147] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3614.154915] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3615.887249] Key type lgssc registered [ 3617.667671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3627.009751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3627.035838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.918365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.709537] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3639.270039] LustreError: 121163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.296135] LustreError: 121163:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3644.387373] LustreError: 121164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.422297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.071247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3651.670899] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3660.373450] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3667.187311] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3674.830377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3675.324301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3682.830939] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3691.949735] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3698.864692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3709.601716] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3713.348460] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3713.356271] Lustre: Skipped 2 previous similar messages [ 3718.055230] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3718.064738] Lustre: Skipped 2 previous similar messages [ 3718.279478] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3718.344940] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3724.951205] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.10@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3724.958413] Lustre: Skipped 2 previous similar messages [ 3734.656513] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.10@tcp 192.168.0.[2,10]@tcp [ 3739.315535] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.10/24@tcp in log lustre-MDT0000 [ 3739.322251] Lustre: Skipped 2 previous similar messages [ 3742.147931] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.10/24@tcp [ 3742.150292] Lustre: Skipped 1 previous similar message [ 3748.326111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.337194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.367678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.369549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.372471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.525626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.541149] Lustre: Skipped 2 previous similar messages [ 3760.628181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3760.638430] Lustre: Skipped 1 previous similar message [ 3762.144473] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3762.400684] Lustre: server umount lustre-OST0000 complete [ 3768.812643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3768.836377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.868371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3770.853071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.698861] Lustre: server umount lustre-MDT0000 complete [ 3775.971159] LustreError: 123606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3775.990195] LustreError: 123606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3776.798357] LustreError: 121145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975180 with bad export cookie 10313521270175671012 [ 3776.805552] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3776.828260] LustreError: 121145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3777.232912] Lustre: server umount lustre-MDT0001 complete [ 3789.657345] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3792.642527] Key type lgssc unregistered [ 3793.007067] LNet: 126024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3793.020634] LNetError: 126024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3793.039589] LNet: Removed LNI 192.168.202.110@tcp [ 3794.017445] Key type .llcrypt unregistered [ 3794.019595] Key type ._llcrypt unregistered [ 3823.484455] Key type ._llcrypt registered [ 3823.487321] Key type .llcrypt registered [ 3823.614138] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3825.152151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3825.278704] alg: No test for adler32 (adler32-zlib) [ 3826.297851] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3826.503793] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3828.151247] Key type lgssc registered [ 3829.892463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3838.273300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3838.299348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.692528] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3839.712495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.722420] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.10/24@tcp [ 3845.382396] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3849.187882] LustreError: 127600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.216706] LustreError: 127600:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3854.306127] LustreError: 127601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3855.430784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.871857] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3855.905679] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.10/24@tcp [ 3855.953338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3860.563327] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3869.861313] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3875.977619] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3883.937775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3884.586343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3889.668944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3892.272255] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3900.356341] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3910.115526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.123961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3910.132682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3910.138426] Lustre: Skipped 1 previous similar message [ 3915.247773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3915.262050] Lustre: Skipped 2 previous similar messages [ 3915.446403] Lustre: server umount lustre-OST0000 complete [ 3920.352649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.364924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.477959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.483249] Lustre: Skipped 2 previous similar messages [ 3926.521741] Lustre: server umount lustre-MDT0000 complete [ 3930.602094] LustreError: 127601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3930.628902] LustreError: 127601:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3930.731590] LustreError: 128341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975334 with bad export cookie 3239642508286248808 [ 3930.738461] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3930.740083] LustreError: 128341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3931.070456] Lustre: server umount lustre-MDT0001 complete [ 3943.164267] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3946.123436] Key type lgssc unregistered [ 3946.426522] LNet: 131188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3946.441658] LNetError: 131188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3947.502720] LNet: Removed LNI 192.168.202.110@tcp [ 3948.930404] Key type .llcrypt unregistered [ 3948.935087] Key type ._llcrypt unregistered [ 3981.233321] Key type ._llcrypt registered [ 3981.245984] Key type .llcrypt registered [ 3981.411409] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 3998.710519] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4000.031672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4000.089222] alg: No test for adler32 (adler32-zlib) [ 4001.345584] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 4001.605719] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4003.351486] Key type lgssc registered [ 4004.951237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4042.827611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4042.860906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4044.125605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4044.146490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4044.239578] Lustre: lustre-MDT0000: new disk, initializing [ 4044.338813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.380762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4049.147448] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4060.439711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.590898] Lustre: 134607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4060.641380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4060.645357] Lustre: Skipped 1 previous similar message [ 4060.726431] Lustre: lustre-MDT0001: new disk, initializing [ 4060.876327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4060.929535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4060.948621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4066.931391] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4078.145422] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4083.580739] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4091.147875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4091.366322] Lustre: lustre-OST0000: new disk, initializing [ 4091.369852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4091.374320] Lustre: 136575:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4091.449084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.706855] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4097.078270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4097.091975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4097.178821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4106.284043] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4110.952916] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4111.177535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4117.153110] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4117.534678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4123.114711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.175120] Lustre: Skipped 1 previous similar message [ 4123.181296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4123.188295] Lustre: Skipped 1 previous similar message [ 4128.231595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4128.259411] Lustre: Skipped 1 previous similar message [ 4133.348052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4133.355206] Lustre: Skipped 1 previous similar message [ 4134.367173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4134.572995] Lustre: server umount lustre-OST0000 complete [ 4138.468511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4138.471720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4138.488459] Lustre: Skipped 1 previous similar message [ 4138.502766] Lustre: Skipped 1 previous similar message [ 4143.584850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.520451] Lustre: server umount lustre-MDT0000 complete [ 4149.443581] LustreError: 134598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975553 with bad export cookie 405966564339322354 [ 4149.446945] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4149.449639] LustreError: 134598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4149.904432] Lustre: server umount lustre-MDT0001 complete [ 4157.121040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.450059] LustreError: 138795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4157.477251] LustreError: 138795:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4157.537156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.606164] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4163.042534] LustreError: 138796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.139895] LustreError: 138795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.341936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.586342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4175.439743] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4189.440491] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4197.725322] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4206.159728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4206.784707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.700077] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4224.277579] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4230.630367] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4230.862985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4237.358350] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4237.596777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4247.672802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.859422] Lustre: lustre-OST0001: new disk, initializing [ 4247.862889] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4247.868438] Lustre: 142169:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4247.956989] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4255.436642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4255.460070] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4255.701511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4255.918940] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4265.593280] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4265.880347] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4271.577553] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4271.758951] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4278.126248] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff913d05e2d000.ost_server_uuid 50 [ 4279.810440] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff913d05e2d000.ost_server_uuid in IDLE state after 0 sec [ 4291.556504] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.557957] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4291.585662] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4291.598840] Lustre: Skipped 2 previous similar messages [ 4296.073465] Lustre: server umount lustre-OST0001 complete [ 4296.674772] LustreError: 140621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4296.728230] LustreError: 140621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4301.798665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.802509] LustreError: 141199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.820081] Lustre: Skipped 2 previous similar messages [ 4301.827310] LustreError: 141199:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4307.085125] Lustre: server umount lustre-OST0000 complete [ 4312.034549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.039623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4312.059289] Lustre: Skipped 1 previous similar message [ 4312.076592] Lustre: Skipped 3 previous similar messages [ 4318.296265] Lustre: server umount lustre-MDT0000 complete [ 4322.276131] LustreError: 138796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.290263] LustreError: 138796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4322.767968] LustreError: 138775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975726 with bad export cookie 405966564339323229 [ 4322.785193] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4322.797561] LustreError: 138775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4323.224128] Lustre: server umount lustre-MDT0001 complete [ 4346.037904] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4349.198626] Key type lgssc unregistered [ 4349.410209] LNet: 145172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4349.416809] LNetError: 145172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4350.444744] LNet: Removed LNI 192.168.202.110@tcp [ 4351.426294] Key type .llcrypt unregistered [ 4351.430962] Key type ._llcrypt unregistered [ 4395.140626] Key type ._llcrypt registered [ 4395.142636] Key type .llcrypt registered [ 4395.274315] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4415.419538] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4416.864500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4417.046348] alg: No test for adler32 (adler32-zlib) [ 4418.233385] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 4418.549438] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4420.235278] Key type lgssc registered [ 4421.705560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4459.847718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4459.887862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.177785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4461.205625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4461.292781] Lustre: lustre-MDT0000: new disk, initializing [ 4461.366983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.381129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4466.018418] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4477.684841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.873581] Lustre: 149343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4477.933622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4477.938409] Lustre: Skipped 1 previous similar message [ 4478.076887] Lustre: lustre-MDT0001: new disk, initializing [ 4478.215708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.253493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4478.277575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4482.745354] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4493.365528] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4499.315964] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4506.738801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4507.199726] Lustre: lustre-OST0000: new disk, initializing [ 4507.204608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4507.210033] Lustre: 151315:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4507.350278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4514.365294] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4515.418333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4515.427948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4515.589270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4524.220428] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4529.044707] Lustre: Failing over lustre-OST0000 [ 4529.162607] Lustre: server umount lustre-OST0000 complete [ 4546.895232] Lustre: 147560:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790975935/real 1790975935] req@000000001889efb5 x1877974263562880/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790975951 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4546.951356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4547.309777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4547.556658] Lustre: 147558:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790975935/real 1790975935] req@00000000c2ba58d3 x1877974263563008/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1790975951 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4547.600672] Lustre: 147558:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4547.868864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4547.890209] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4552.720899] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4552.869958] Lustre: 147557:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790975940/real 1790975940] req@000000007542d2b3 x1877974263564672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790975956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4552.931574] Lustre: 147557:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4556.195453] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4557.002234] Lustre: 147559:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790975945/real 1790975945] req@000000003dd63a60 x1877974263565952/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790975961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4557.025731] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4557.025943] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4560.788104] Lustre: Failing over lustre-MDT0000 [ 4562.402030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.414447] LustreError: 151160:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.429843] Lustre: Skipped 2 previous similar messages [ 4562.454456] LustreError: 151160:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4562.989541] LustreError: 151160:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.092591] Lustre: server umount lustre-MDT0000 complete [ 4563.487146] Lustre: 147559:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790975950/real 1790975950] req@000000008d9ac1cc x1877974263566976/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790975966 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4563.501885] Lustre: 147559:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4567.527595] LustreError: 152235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.538528] LustreError: 152235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4572.646039] LustreError: 149351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4572.679536] LustreError: 149351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4577.771787] LustreError: 149354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.797169] LustreError: 149354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4578.784088] Lustre: 147558:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790975966/real 1790975966] req@00000000e24f7e4e x1877974263576064/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790975982 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4578.799544] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4584.293393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.007786] LustreError: 149349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.057500] LustreError: 149349:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 4588.421293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.505481] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4589.581862] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4593.523806] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4593.656476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4593.666946] Lustre: Skipped 1 previous similar message [ 4593.705812] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4593.801971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4603.361362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4603.370432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4603.382295] Lustre: Skipped 1 previous similar message [ 4603.388802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4603.929509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4608.995026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4609.011526] Lustre: Skipped 1 previous similar message [ 4609.382332] Lustre: server umount lustre-OST0000 complete [ 4614.112354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4614.114572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.129568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.131891] Lustre: Skipped 2 previous similar messages [ 4614.147580] Lustre: Skipped 1 previous similar message [ 4619.233293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4619.242467] Lustre: Skipped 1 previous similar message [ 4620.136188] Lustre: server umount lustre-MDT0000 complete [ 4624.373748] LustreError: 149351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4624.397048] LustreError: 149351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4625.728162] LustreError: 149335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976029 with bad export cookie 10744560184285673122 [ 4625.731605] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4625.739059] LustreError: 149335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4626.382956] Lustre: server umount lustre-MDT0001 complete [ 4640.100595] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4643.287312] Key type lgssc unregistered [ 4643.541925] LNet: 154971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4643.554682] LNetError: 154971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4643.580994] LNet: Removed LNI 192.168.202.110@tcp [ 4644.810799] Key type .llcrypt unregistered [ 4644.813067] Key type ._llcrypt unregistered [ 4686.059317] Key type ._llcrypt registered [ 4686.065171] Key type .llcrypt registered [ 4686.165735] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4687.395580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4687.421291] alg: No test for adler32 (adler32-zlib) [ 4688.770129] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 4689.071838] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4690.831266] Key type lgssc registered [ 4692.347859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4701.116842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4701.134881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.698735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4708.374425] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4711.909302] LustreError: 156507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.936406] LustreError: 156507:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4716.879219] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4717.024947] LustreError: 156508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.147616] LustreError: 156507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.167528] LustreError: 156536:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4722.188706] LustreError: 156536:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4722.197602] LustreError: 156536:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4722.621942] Lustre: server umount lustre-MDT0000 complete [ 4743.203256] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4753.962283] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4790.129875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.441487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4790.500612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4790.627557] Lustre: lustre-MDT0000: new disk, initializing [ 4790.723503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.747158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4796.307193] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4808.221952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4808.317731] Lustre: 159840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4808.365316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4808.370266] Lustre: Skipped 1 previous similar message [ 4808.475719] Lustre: lustre-MDT0001: new disk, initializing [ 4808.591240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4808.616494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4808.627439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4813.155887] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4823.056540] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4830.393952] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4838.436468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4838.805655] Lustre: lustre-OST0000: new disk, initializing [ 4838.814720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4838.819910] Lustre: 161813:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4838.946133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.491285] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4845.659884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4845.672174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4845.800940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4855.661478] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4860.214775] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4862.310457] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:24:26 (1790976266) [ 5096.929099] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.933193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.948491] Lustre: Skipped 1 previous similar message [ 5096.961448] Lustre: Skipped 2 previous similar messages [ 5102.056707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5102.061585] Lustre: Skipped 2 previous similar messages [ 5103.137499] Lustre: server umount lustre-MDT0000 complete [ 5107.182747] LustreError: 159849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.193682] LustreError: 159835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976511 with bad export cookie 6185468603360087890 [ 5107.202609] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5107.203490] LustreError: 159849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5107.249348] LustreError: 159835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5112.293042] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.295216] LustreError: 159847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5112.305075] Lustre: Skipped 1 previous similar message [ 5112.339440] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5113.772824] Lustre: server umount lustre-MDT0001 complete [ 5131.231083] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5131.364375] Lustre: server umount lustre-OST0000 complete [ 5140.815687] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5150.643131] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5187.061409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5187.333530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5187.399212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5187.519836] Lustre: lustre-MDT0000: new disk, initializing [ 5187.651228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.677345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5192.612858] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5203.758329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5203.892214] Lustre: 166483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5203.954373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5203.964779] Lustre: Skipped 1 previous similar message [ 5204.092627] Lustre: lustre-MDT0001: new disk, initializing [ 5204.222950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5204.293446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5204.304146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5209.070713] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5219.126996] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5225.618309] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5232.617216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5232.947366] Lustre: lustre-OST0000: new disk, initializing [ 5232.951350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5232.961803] Lustre: 168451:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5233.053864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5238.867594] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5240.365666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5240.371152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5240.454622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5249.455030] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5256.287535] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5256.560196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5261.785523] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5261.964342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5265.892103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5265.905522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.947874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5266.400966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5270.486194] Lustre: server umount lustre-OST0000 complete [ 5276.128858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5276.144894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.172265] Lustre: Skipped 1 previous similar message [ 5276.182235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5276.191893] Lustre: Skipped 2 previous similar messages [ 5281.760463] LustreError: 166494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.777510] LustreError: 166494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5281.818358] Lustre: server umount lustre-MDT0000 complete [ 5285.975945] LustreError: 167279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976690 with bad export cookie 6185468603360246720 [ 5285.995462] LustreError: 167279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5285.997041] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5286.546409] Lustre: server umount lustre-MDT0001 complete [ 5295.096934] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:31:39 (1790976699) [ 5313.033767] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5326.578252] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5371.725318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.052283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5372.095276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5372.200839] Lustre: lustre-MDT0000: new disk, initializing [ 5372.309651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5372.351690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5376.525304] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5388.299485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.433747] Lustre: 173207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5388.624085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5393.440065] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5403.543848] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5410.323381] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5417.520893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5417.830550] Lustre: lustre-OST0000: new disk, initializing [ 5417.835513] Lustre: Skipped 1 previous similar message [ 5417.844949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5417.848837] Lustre: Skipped 2 previous similar messages [ 5417.853110] Lustre: 175186:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5424.238616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5424.250243] Lustre: Skipped 1 previous similar message [ 5424.258284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5424.375060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5424.764157] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5435.442485] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5445.098862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.107125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5445.107280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5445.117753] Lustre: Skipped 2 previous similar messages [ 5445.139996] Lustre: Skipped 2 previous similar messages [ 5449.749519] Lustre: server umount lustre-OST0000 complete [ 5455.329404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.215228] Lustre: server umount lustre-MDT0000 complete [ 5460.450239] LustreError: 173220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5460.479150] LustreError: 173220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5465.039544] LustreError: 173202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976869 with bad export cookie 6185468603360247602 [ 5465.041762] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5465.059862] LustreError: 173202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5466.162517] Lustre: server umount lustre-MDT0001 complete [ 5479.380996] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:34:42 (1790976882) [ 5498.840355] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5510.584706] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5552.854317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.126546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5553.208557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5553.354271] Lustre: lustre-MDT0000: new disk, initializing [ 5553.538530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.543458] Lustre: Skipped 2 previous similar messages [ 5553.566995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5558.501900] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5571.274497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.489803] Lustre: 179512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5571.884040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5578.302452] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5589.327993] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5597.823557] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5606.999667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5607.353305] Lustre: 181489:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5615.276611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5615.517355] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5615.537481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5627.374238] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5636.065425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5636.075762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5636.084021] LustreError: Skipped 1 previous similar message [ 5636.084690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5636.084695] Lustre: Skipped 2 previous similar messages [ 5636.145956] Lustre: Skipped 2 previous similar messages [ 5641.546973] Lustre: server umount lustre-OST0000 complete [ 5660.133246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5660.591621] Lustre: server umount lustre-MDT0000 complete [ 5661.668180] LustreError: 179524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.690523] LustreError: 179524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5665.066524] LustreError: 179505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977069 with bad export cookie 6185468603360248687 [ 5665.072643] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5665.076811] LustreError: 179505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5665.699545] Lustre: server umount lustre-MDT0001 complete [ 5678.248420] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5680.996242] Key type lgssc unregistered [ 5681.464272] LNet: 183347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5681.476593] LNetError: 183347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5681.497401] LNet: Removed LNI 192.168.202.110@tcp [ 5682.503546] Key type .llcrypt unregistered [ 5682.509316] Key type ._llcrypt unregistered [ 5710.987773] Key type ._llcrypt registered [ 5710.992783] Key type .llcrypt registered [ 5711.148154] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5713.071302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5713.118753] alg: No test for adler32 (adler32-zlib) [ 5714.202500] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 5714.515244] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 5716.215211] Key type lgssc registered [ 5717.979845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5727.490259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5727.529839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.151782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5735.312033] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5739.505234] LustreError: 184781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.525652] LustreError: 184781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5744.612042] LustreError: 184780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.954323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.322085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5751.373877] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5759.223614] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5766.019744] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5773.898872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5774.187288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5775.227589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5780.820896] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5789.496180] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5811.172148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5811.184568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.200699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5812.193844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5812.210695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.792411] Lustre: server umount lustre-OST0000 complete [ 5822.440085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.443756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5822.452245] Lustre: Skipped 1 previous similar message [ 5822.458973] Lustre: Skipped 1 previous similar message [ 5825.464468] Lustre: server umount lustre-MDT0000 complete [ 5827.558285] LustreError: 184776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.587861] LustreError: 184776:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5829.972904] LustreError: 184763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977234 with bad export cookie 18328753603704993367 [ 5829.987490] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5830.331732] Lustre: server umount lustre-MDT0001 complete [ 5841.978638] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5844.450113] Key type lgssc unregistered [ 5844.692691] LNet: 188337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5844.697080] LNetError: 188337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5844.709421] LNet: Removed LNI 192.168.202.110@tcp [ 5845.496633] Key type .llcrypt unregistered [ 5845.499984] Key type ._llcrypt unregistered [ 5870.844765] Key type ._llcrypt registered [ 5870.850497] Key type .llcrypt registered [ 5871.062786] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5872.597548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5872.834422] alg: No test for adler32 (adler32-zlib) [ 5874.204752] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 5874.495092] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 5876.159277] Key type lgssc registered [ 5877.843172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5886.965490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5887.000288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5888.697264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.773658] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5898.211230] LustreError: 189772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5898.236524] LustreError: 189772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5903.027283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5903.328353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5903.487770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5908.953934] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5918.560846] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5925.237697] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5933.338170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5933.540798] LustreError: 191591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.693470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5939.196346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5940.909703] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5950.548350] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5959.658127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5959.677981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.702353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.745563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.755587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.769565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5965.021512] Lustre: server umount lustre-OST0000 complete [ 5972.563819] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff913d05171800.ost_server_uuid 50 [ 5974.590607] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5990.368131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5990.380603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.399468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5990.410102] Lustre: Skipped 2 previous similar messages [ 5994.465629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.467105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5994.491237] Lustre: Skipped 1 previous similar message [ 5996.754902] Lustre: server umount lustre-MDT0000 complete [ 5999.588884] LustreError: 189766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.606337] LustreError: 189766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6000.826925] LustreError: 189753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977404 with bad export cookie 3373411562578833561 [ 6000.831024] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6000.832457] LustreError: 189753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6001.297238] Lustre: server umount lustre-MDT0001 complete [ 6012.002042] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:43:35 (1790977415) [ 6021.323437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.155618] LustreError: 193687:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.278831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.749254] LustreError: 193688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6028.047601] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6036.976957] LustreError: 193688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.024265] LustreError: 193688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6038.240975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6038.587121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6044.990151] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6054.128851] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6060.457653] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6068.555176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6068.842709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.477134] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6086.029174] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6092.419074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6094.127814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6094.514475] Lustre: lustre-OST0001: new disk, initializing [ 6094.521625] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6094.538584] Lustre: 196531:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6094.668625] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6102.302272] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6103.663170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6103.690239] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6103.867137] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6113.384660] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6118.368594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6118.377173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.395384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6123.065353] Lustre: server umount lustre-OST0000 complete [ 6124.523153] LustreError: 196437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6124.550543] LustreError: 196437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6130.934873] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6131.216253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6137.151118] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6137.387782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6155.234319] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6155.240085] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6155.257763] Lustre: Skipped 3 previous similar messages [ 6155.258410] Lustre: Skipped 2 previous similar messages [ 6157.142635] Lustre: server umount lustre-OST0001 complete [ 6176.224711] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6176.533770] Lustre: server umount lustre-MDT0000 complete [ 6180.839338] LustreError: 194416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6180.859444] LustreError: 194416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [ 6181.099626] LustreError: 193670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977585 with bad export cookie 3373411562578834632 [ 6181.127979] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6181.128682] LustreError: 193670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6182.218730] Lustre: server umount lustre-MDT0001 complete [ 6213.059414] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:46:56 (1790977616) [ 6220.705604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.024452] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6221.051817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6221.323867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6226.750719] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6238.239918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.398626] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6238.492950] Lustre: 199971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6245.140184] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6255.305847] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6263.007781] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6271.435223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.738876] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6277.908935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6280.110508] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6288.915582] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6296.852275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6296.925047] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6297.263959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6297.268598] Lustre: Skipped 2 previous similar messages [ 6305.285767] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6314.437366] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6318.568379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.602321] Lustre: Skipped 2 previous similar messages [ 6318.617896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6318.622560] Lustre: Skipped 8 previous similar messages [ 6323.415324] Lustre: server umount lustre-OST0000 complete [ 6323.708339] LustreError: 202867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6344.161258] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6344.178631] LustreError: Skipped 1 previous similar message [ 6347.709986] Lustre: server umount lustre-OST0001 complete [ 6354.400285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.406888] Lustre: Skipped 4 previous similar messages [ 6358.673057] Lustre: server umount lustre-MDT0000 complete [ 6363.030027] LustreError: 199963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977767 with bad export cookie 3373411562578836039 [ 6363.040465] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6363.043138] LustreError: 199963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6363.507442] Lustre: server umount lustre-MDT0001 complete [ 6394.487384] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:49:57 (1790977797) [ 6417.452621] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 6429.694872] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 6470.375623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.693632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6470.743804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6470.965532] Lustre: lustre-MDT0000: new disk, initializing [ 6471.158802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6471.175498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6476.161796] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6489.128372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.332968] Lustre: 208076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6489.430242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6489.433970] Lustre: Skipped 1 previous similar message [ 6489.495473] Lustre: lustre-MDT0001: new disk, initializing [ 6489.712740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6489.738774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6495.565728] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6507.934315] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6516.169893] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6525.026503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6525.331217] Lustre: lustre-OST0000: new disk, initializing [ 6525.334487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6525.338784] Lustre: 210047:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6533.127163] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6535.269855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6535.281425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6535.403693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6543.655710] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6549.311360] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6549.658886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6555.445098] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6555.730036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6561.250596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6561.257106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6561.260318] LustreError: Skipped 1 previous similar message [ 6561.260900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6561.260903] Lustre: Skipped 5 previous similar messages [ 6561.305796] Lustre: Skipped 1 previous similar message [ 6563.984189] Lustre: server umount lustre-OST0000 complete [ 6574.326635] Lustre: server umount lustre-MDT0000 complete [ 6576.609730] LustreError: 208112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6576.641714] LustreError: 208112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 6579.124874] LustreError: 208067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977983 with bad export cookie 3373411562578837600 [ 6579.133189] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6579.362858] Lustre: server umount lustre-MDT0001 complete [ 6587.439615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.555219] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6607.034729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.504882] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6626.035946] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6634.848260] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6644.526326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6652.953735] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6661.405804] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6666.859710] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6667.162631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6672.267845] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6672.517132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6677.985257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6678.000335] LustreError: Skipped 1 previous similar message [ 6681.100443] Lustre: server umount lustre-OST0000 complete [ 6688.173243] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6693.715951] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6723.766248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6731.849464] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6743.471911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6743.504421] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6743.506716] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6743.537609] Lustre: Skipped 1 previous similar message [ 6744.774337] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6748.063137] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6758.880638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6758.886446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6758.893359] LustreError: Skipped 1 previous similar message [ 6758.907669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6758.910646] Lustre: Skipped 5 previous similar messages [ 6758.934355] Lustre: Skipped 7 previous similar messages [ 6762.554927] Lustre: server umount lustre-OST0000 complete [ 6772.712191] Lustre: server umount lustre-MDT0000 complete [ 6776.319624] LustreError: 212245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978180 with bad export cookie 3373411562578838482 [ 6776.324112] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6776.337651] LustreError: 212245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6776.703244] Lustre: server umount lustre-MDT0001 complete [ 6786.784694] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:56:30 (1790978190) [ 6793.634990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6794.451993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6794.469109] Lustre: Skipped 6 previous similar messages [ 6799.634954] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6808.882025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.646914] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6825.139924] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6831.422540] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6838.557502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6845.917714] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6853.615724] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6859.229618] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6859.489166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6864.999103] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6865.361581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6873.232651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.534237] Lustre: lustre-OST0001: new disk, initializing [ 6873.542334] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6873.551069] Lustre: 221608:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6880.731134] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6880.875476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6880.893177] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6880.975882] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6891.428492] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6895.962275] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6896.136922] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6901.133530] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6901.421929] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6906.881937] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6906.907704] LustreError: Skipped 2 previous similar messages [ 6909.978765] Lustre: server umount lustre-OST0001 complete [ 6911.977660] LustreError: 221616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6911.997737] LustreError: 221616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6918.300430] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6918.657692] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6925.061012] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6925.353843] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6955.244956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6961.141687] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6961.159612] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6961.162982] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6961.185743] Lustre: Skipped 1 previous similar message [ 6962.234416] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6971.100413] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6972.877457] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6981.828427] Lustre: server umount lustre-OST0001 complete [ 7005.128896] Lustre: server umount lustre-OST0000 complete [ 7016.170328] Lustre: server umount lustre-MDT0000 complete [ 7020.337388] LustreError: 221606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978424 with bad export cookie 3373411562578839448 [ 7020.341880] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7020.348130] LustreError: 221606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7026.901898] Lustre: server umount lustre-MDT0001 complete [ 7053.721183] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:00:57 (1790978457) [ 7065.303490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.816948] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7065.891749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7072.160899] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7081.507567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.654570] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7081.699085] Lustre: 226867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7087.475695] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7095.881990] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7102.221924] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7109.906530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7110.361344] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7119.336344] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7128.762538] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7139.320568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7139.425556] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7146.626322] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7153.706838] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7155.794731] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7161.189217] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7161.639412] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7168.807868] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7169.044408] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7174.507444] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff913d05859800.ost_server_uuid 50 [ 7175.914212] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff913d05859800.ost_server_uuid in IDLE state after 0 sec [ 7176.857413] Lustre: Permanently deactivating lustre-OST0001 [ 7176.867588] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7180.332691] Lustre: Permanently reactivating lustre-OST0001 [ 7185.892846] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7185.915170] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.916183] LustreError: Skipped 2 previous similar messages [ 7185.928185] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7185.940607] Lustre: Skipped 11 previous similar messages [ 7185.968015] Lustre: Skipped 15 previous similar messages [ 7191.122545] Lustre: server umount lustre-OST0001 complete [ 7201.202452] Lustre: server umount lustre-OST0000 complete [ 7212.480631] Lustre: server umount lustre-MDT0000 complete [ 7217.454203] LustreError: 226860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978621 with bad export cookie 3373411562578840736 [ 7217.462034] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7217.474797] LustreError: 226860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7218.047693] Lustre: server umount lustre-MDT0001 complete [ 7243.803505] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:04:07 (1790978647) [ 7252.809604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.224362] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7253.245310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7253.250993] Lustre: Skipped 5 previous similar messages [ 7258.261919] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7267.507458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.619961] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7267.667108] Lustre: 233573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7273.806240] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7283.421176] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7290.750700] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7299.159293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7299.401250] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7299.437097] Lustre: Permanently deactivating lustre-OST0000 [ 7299.480610] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7306.818270] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7307.174427] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7315.513179] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7323.293237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7323.406949] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7323.756413] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7323.780142] Lustre: Skipped 11 previous similar messages [ 7331.681394] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7332.178840] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7341.885774] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7343.994176] Lustre: Permanently reactivating lustre-OST0000 [ 7344.009476] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7344.022854] Lustre: Skipped 2 previous similar messages [ 7351.933611] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7351.954480] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 0000000013330b64, cur 1790978756 deadline 1790978741 last 1790978711 [ 7352.010939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7374.304092] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7374.471106] Lustre: server umount lustre-OST0001 complete [ 7393.247206] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7393.552239] Lustre: server umount lustre-OST0000 complete [ 7404.782515] Lustre: server umount lustre-MDT0000 complete [ 7409.645833] LustreError: 237170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978813 with bad export cookie 3373411562578842409 [ 7409.660084] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.663314] LustreError: 237170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7410.153606] Lustre: server umount lustre-MDT0001 complete [ 7424.650804] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 7428.807411] Key type lgssc unregistered [ 7429.360618] LNet: 238456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7429.379290] LNetError: 238456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7429.412514] LNet: Removed LNI 192.168.202.110@tcp [ 7430.505078] Key type .llcrypt unregistered [ 7430.510992] Key type ._llcrypt unregistered [ 7456.686802] Key type ._llcrypt registered [ 7456.700810] Key type .llcrypt registered [ 7456.935280] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7458.357735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7458.386764] alg: No test for adler32 (adler32-zlib) [ 7459.696699] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 7460.130027] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 7461.944604] Key type lgssc registered [ 7463.651685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7474.572907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7474.588651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.309048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7481.623526] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7486.452172] LustreError: 239941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.481396] LustreError: 239941:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7491.574308] LustreError: 239942:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.746186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7491.876961] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7491.912762] Lustre: 239930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7491.939129] Lustre: Permanently deactivating lustre-MDT0001 [ 7491.957845] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7492.027582] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7492.035474] LustreError: 239972:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7492.047262] LustreError: 239972:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7492.065043] LustreError: 239972:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7492.301563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7498.311129] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7507.472192] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7514.304668] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7521.396218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7521.724365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.257187] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7537.068171] LustreError: 241784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.100752] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7537.546522] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7542.261862] LustreError: 241784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7544.917686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7545.152803] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7546.198629] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7546.205194] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7546.298048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7552.429374] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7561.319217] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7564.460027] Lustre: Permanently reactivating lustre-MDT0001 [ 7564.499508] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7564.507744] Lustre: Skipped 2 previous similar messages [ 7569.494910] Lustre: lustre-MDT0001: Client f565b3b0-0b68-43dd-b1d3-3cd617a220de (at 192.168.202.10@tcp) reconnecting [ 7577.411282] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7579.401643] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7585.673495] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7585.937806] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7591.385975] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7591.621371] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7594.548929] Lustre: Permanently deactivating lustre-MDT0001 [ 7594.567789] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7594.571663] Lustre: Skipped 2 previous similar messages [ 7602.833035] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7612.899333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7612.909031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7612.919096] Lustre: Skipped 2 previous similar messages [ 7612.933849] Lustre: Skipped 3 previous similar messages [ 7618.019638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7618.029685] Lustre: Skipped 3 previous similar messages [ 7618.620573] Lustre: server umount lustre-MDT0000 complete [ 7622.564302] LustreError: 239923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979026 with bad export cookie 9399282387670323567 [ 7622.571667] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7622.577054] LustreError: 239923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7622.817123] Lustre: server umount lustre-MDT0001 complete [ 7628.623455] Lustre: server umount lustre-OST0000 complete [ 7640.326558] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:10:44 (1790979044) [ 7651.933298] Lustre: server umount lustre-OST0001 complete [ 7659.746070] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 7672.373711] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7714.089750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.435852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7714.446197] Lustre: Skipped 2 previous similar messages [ 7714.493781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7714.616605] Lustre: lustre-MDT0000: new disk, initializing [ 7714.714865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7714.735694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7720.044740] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7732.566801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.706508] Lustre: 248229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7732.745894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7732.749456] Lustre: Skipped 1 previous similar message [ 7732.812578] Lustre: lustre-MDT0001: new disk, initializing [ 7732.891131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7732.928822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7732.938361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7738.093679] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7748.385959] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7754.337992] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7762.038188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7762.366567] Lustre: lustre-OST0000: new disk, initializing [ 7762.379890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7762.394036] Lustre: 250200:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7762.498271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7770.041541] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7770.692151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7770.714325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7770.847980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7780.822660] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7783.667143] LustreError: 248237:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7793.052940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7793.228113] Lustre: lustre-OST0001: new disk, initializing [ 7793.232589] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7793.239468] Lustre: 251315:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7793.744713] LustreError: 248237:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7793.770476] LustreError: 251268:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7801.055676] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7801.669432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7801.692508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7801.844877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7803.823351] LustreError: 251268:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7813.040577] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7814.729872] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7817.184383] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7817.192697] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7817.206553] Lustre: Skipped 1 previous similar message [ 7817.220452] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7817.232439] Lustre: Skipped 1 previous similar message [ 7822.311189] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7822.319041] Lustre: Skipped 1 previous similar message [ 7823.288741] Lustre: server umount lustre-OST0001 complete [ 7827.432523] LustreError: 250185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.451173] LustreError: 250185:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7837.666987] LustreError: 250186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.705648] LustreError: 250186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7842.784858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7842.799209] LustreError: Skipped 1 previous similar message [ 7842.801682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7842.815784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7842.823237] Lustre: Skipped 2 previous similar messages [ 7842.851431] Lustre: Skipped 1 previous similar message [ 7853.026255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7853.029881] Lustre: Skipped 3 previous similar messages [ 7856.096980] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7856.504806] Lustre: server umount lustre-OST0000 complete [ 7861.217291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7861.222808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7867.274827] Lustre: server umount lustre-MDT0000 complete [ 7868.402644] LustreError: 249393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7868.434032] LustreError: 249393:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7871.630450] LustreError: 248221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979275 with bad export cookie 9399282387670326381 [ 7871.631256] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7871.641623] LustreError: 248221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7878.205727] Lustre: server umount lustre-MDT0001 complete [ 7892.537189] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 7895.850910] Key type lgssc unregistered [ 7896.436534] LNet: 253347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7896.452500] LNetError: 253347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7896.470219] LNet: Removed LNI 192.168.202.110@tcp [ 7897.942863] Key type .llcrypt unregistered [ 7897.944216] Key type ._llcrypt unregistered [ 7944.772765] Key type ._llcrypt registered [ 7944.775512] Key type .llcrypt registered [ 7944.909749] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7947.149297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7947.182580] alg: No test for adler32 (adler32-zlib) [ 7948.480429] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 7948.835919] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 7950.615292] Key type lgssc registered [ 7951.885979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7960.575779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7960.606507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7961.949882] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7962.003822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7962.257444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7967.232746] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7976.550389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.710115] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7976.748536] Lustre: 255573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7977.055142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7982.031574] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7991.858973] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7998.493672] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8005.658174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8005.853825] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8006.039123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8011.873821] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8014.579367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8020.753461] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8080.399312] Lustre: server umount lustre-OST0000 complete [ 8085.389595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8097.759117] Lustre: 254886:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790979485/real 1790979485] req@00000000ed13d3ab x1877977966098048/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790979501 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8097.787616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8098.430762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8098.957900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8102.031233] Lustre: 254887:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790979490/real 1790979490] req@000000004f2e2de8 x1877977966099968/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790979506 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8102.062830] Lustre: 254887:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 8106.049033] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8107.039164] Lustre: 254888:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790979495/real 1790979495] req@000000008b47d9ff x1877977966101248/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790979511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8107.059490] Lustre: 254888:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8107.126255] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8107.159447] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8108.216176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8108.243562] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8116.507570] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8158.688693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8158.698755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.722706] Lustre: Skipped 1 previous similar message [ 8158.743277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8159.718148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8159.719343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8159.734511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.882478] Lustre: server umount lustre-OST0000 complete [ 8169.954069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8169.965133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8169.967548] Lustre: Skipped 1 previous similar message [ 8175.079055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8175.090168] Lustre: Skipped 2 previous similar messages [ 8180.201930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8180.215833] Lustre: Skipped 1 previous similar message [ 8183.264732] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8183.481561] Lustre: server umount lustre-MDT0000 complete [ 8187.535521] LustreError: 255565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979591 with bad export cookie 491261714032755885 [ 8187.539955] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8187.545511] LustreError: 255565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8187.943483] Lustre: server umount lustre-MDT0001 complete [ 8201.435053] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8205.362787] Key type lgssc unregistered [ 8205.822753] LNet: 260995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8205.833520] LNetError: 260995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8205.858910] LNet: Removed LNI 192.168.202.110@tcp [ 8206.825616] Key type .llcrypt unregistered [ 8206.828096] Key type ._llcrypt unregistered [ 8231.152987] Key type ._llcrypt registered [ 8231.157417] Key type .llcrypt registered [ 8231.351978] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8232.687092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8232.708062] alg: No test for adler32 (adler32-zlib) [ 8233.793328] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 8234.072424] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8235.759372] Key type lgssc registered [ 8236.951480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8245.025658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8245.042809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.539891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.557533] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8255.993345] LustreError: 262428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8256.007535] LustreError: 262428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8260.396675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8260.851170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8265.441839] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8273.786479] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8279.909263] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8286.061807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.438308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8291.565474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8293.609688] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8300.615629] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8330.720276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8330.730114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8330.743437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8330.775384] Lustre: Skipped 1 previous similar message [ 8335.749094] Lustre: server umount lustre-OST0000 complete [ 8340.961827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.967874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8340.977374] Lustre: Skipped 1 previous similar message [ 8340.991050] Lustre: Skipped 2 previous similar messages [ 8345.728432] Lustre: server umount lustre-MDT0000 complete [ 8346.080827] LustreError: 262429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8346.094241] LustreError: 262429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8349.580237] LustreError: 262409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979753 with bad export cookie 17662654907472366267 [ 8349.582128] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8349.584778] LustreError: 262409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8350.081744] Lustre: server umount lustre-MDT0001 complete [ 8364.055467] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8367.644749] Key type lgssc unregistered [ 8368.021601] LNet: 266736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8368.026359] LNetError: 266736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8368.036973] LNet: Removed LNI 192.168.202.110@tcp [ 8369.241488] Key type .llcrypt unregistered [ 8369.250103] Key type ._llcrypt unregistered [ 8388.310621] Key type ._llcrypt registered [ 8388.312962] Key type .llcrypt registered [ 8388.407685] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8389.841636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8389.953546] alg: No test for adler32 (adler32-zlib) [ 8391.070097] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 8391.329509] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8393.087284] Key type lgssc registered [ 8394.379092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8403.754112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8403.772978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.179685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.884150] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8415.714274] LustreError: 267975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8415.726547] LustreError: 267975:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8417.933623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8418.343517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8422.679796] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8430.600644] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8436.514354] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8443.325204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8443.603800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8449.793470] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8450.798903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8458.325531] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8471.522644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.524125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8471.551334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8471.570738] Lustre: Skipped 1 previous similar message [ 8476.643786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8476.661141] Lustre: Skipped 1 previous similar message [ 8476.970987] Lustre: server umount lustre-OST0000 complete [ 8484.833443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8484.846709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8484.867300] Lustre: Skipped 1 previous similar message [ 8484.878337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8486.881428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8486.883494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.286547] Lustre: server umount lustre-MDT0000 complete [ 8492.001238] LustreError: 267974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.030671] LustreError: 267974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8494.272567] LustreError: 267956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979898 with bad export cookie 2050207547912766214 [ 8494.284836] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8494.976920] Lustre: server umount lustre-MDT0001 complete [ 8508.157855] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8510.733305] Key type lgssc unregistered [ 8511.120187] LNet: 271654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8511.129234] LNetError: 271654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8511.148732] LNet: Removed LNI 192.168.202.110@tcp [ 8512.136966] Key type .llcrypt unregistered [ 8512.139118] Key type ._llcrypt unregistered [ 8538.089921] Key type ._llcrypt registered [ 8538.092174] Key type .llcrypt registered [ 8538.183500] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8539.128878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8539.339597] alg: No test for adler32 (adler32-zlib) [ 8540.599874] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 8540.937570] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8542.759179] Key type lgssc registered [ 8545.235024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8554.181947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8554.207306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.597265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8561.884280] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8565.730382] LustreError: 273088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8565.754175] LustreError: 273088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8566.758646] LustreError: 273087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8571.755487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8571.871976] LustreError: 273088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8572.311281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8576.659809] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8584.709308] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8590.781922] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8597.285848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8597.733181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8601.854459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8604.176483] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8611.606344] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8637.921859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8637.928373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8637.928837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8637.939138] Lustre: Skipped 1 previous similar message [ 8637.961384] Lustre: Skipped 1 previous similar message [ 8643.043359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8644.204470] Lustre: server umount lustre-OST0000 complete [ 8648.162676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.180156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8648.190706] Lustre: Skipped 2 previous similar messages [ 8653.282731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8653.289506] Lustre: Skipped 1 previous similar message [ 8654.285199] Lustre: server umount lustre-MDT0000 complete [ 8658.139574] LustreError: 273067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980062 with bad export cookie 16326788227183384124 [ 8658.153727] LustreError: 273067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8658.154267] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8658.459963] Lustre: server umount lustre-MDT0001 complete [ 8669.873673] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8672.683961] Key type lgssc unregistered [ 8673.053764] LNet: 277393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8673.064912] LNetError: 277393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8673.078625] LNet: Removed LNI 192.168.202.110@tcp [ 8673.861210] Key type .llcrypt unregistered [ 8673.863233] Key type ._llcrypt unregistered [ 8694.433362] Key type ._llcrypt registered [ 8694.435321] Key type .llcrypt registered [ 8694.590413] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8696.403208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8696.446974] alg: No test for adler32 (adler32-zlib) [ 8697.681323] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 8698.179359] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8700.015272] Key type lgssc registered [ 8701.675134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8711.107616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8711.154384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8713.065369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.744148] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8723.434012] LustreError: 278646:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8723.461057] LustreError: 278646:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8728.552334] LustreError: 278648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8729.376700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8729.775169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8735.171175] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8743.440549] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8750.205336] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8757.829227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8758.064544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8759.105465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8764.945668] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8772.530171] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8789.995874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8790.004288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8790.008353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8790.019317] Lustre: Skipped 1 previous similar message [ 8792.300230] Lustre: server umount lustre-OST0000 complete [ 8796.131832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8796.143568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8796.157824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8796.163586] Lustre: Skipped 1 previous similar message [ 8800.223774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8800.226937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.061877] Lustre: server umount lustre-MDT0000 complete [ 8805.349864] LustreError: 278640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8805.397338] LustreError: 278640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8805.875085] LustreError: 278612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980210 with bad export cookie 17767043967765910856 [ 8805.881217] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8805.885239] LustreError: 278612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8806.160652] Lustre: server umount lustre-MDT0001 complete [ 8817.604669] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8821.343814] Key type lgssc unregistered [ 8821.657928] LNet: 282346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8821.680956] LNetError: 282346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8821.702541] LNet: Removed LNI 192.168.202.110@tcp [ 8822.547701] Key type .llcrypt unregistered [ 8822.551290] Key type ._llcrypt unregistered [ 8835.850239] Key type ._llcrypt registered [ 8835.854633] Key type .llcrypt registered [ 8836.004713] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8856.342473] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 8874.564001] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8875.900980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8876.006117] alg: No test for adler32 (adler32-zlib) [ 8877.135228] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 8877.430295] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8879.159212] Key type lgssc registered [ 8880.975704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8917.532989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8917.557033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.891357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8918.928247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8918.975751] Lustre: lustre-MDT0000: new disk, initializing [ 8919.027869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.041353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8923.821942] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8934.637467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8934.780627] Lustre: 285917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8934.851913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8934.862706] Lustre: Skipped 1 previous similar message [ 8934.948639] Lustre: lustre-MDT0001: new disk, initializing [ 8935.084961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8935.140249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8935.155960] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8940.396793] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8949.210492] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8954.800220] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8961.849722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8962.098346] Lustre: lustre-OST0000: new disk, initializing [ 8962.103702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8962.115725] Lustre: 287888:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8962.265286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8968.095355] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8971.846719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8971.867281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8971.967831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8977.599192] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8981.930475] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8982.184658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8987.152741] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8987.354386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8992.739871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8992.753328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8992.771068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8992.783274] Lustre: Skipped 1 previous similar message [ 8995.586675] Lustre: server umount lustre-OST0000 complete [ 9001.953765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9001.965743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.977729] Lustre: Skipped 1 previous similar message [ 9001.982541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9002.980542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.004890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9003.014917] Lustre: Skipped 1 previous similar message [ 9005.758592] Lustre: server umount lustre-MDT0000 complete [ 9009.312302] LustreError: 285911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980413 with bad export cookie 10746483154107637771 [ 9009.330925] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9009.671460] Lustre: server umount lustre-MDT0001 complete [ 9018.857663] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:33:42 (1790980422) [ 9026.223795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9026.710420] LustreError: 290300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9026.743405] LustreError: 290300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9026.792265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9031.608978] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9032.168792] LustreError: 290301:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9036.259531] LustreError: 290300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9040.637568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9040.996100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9045.578187] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9053.716147] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9059.901672] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9066.992105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9067.471693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9074.321368] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9083.574739] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9096.162418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9096.173678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9096.178564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9096.189316] Lustre: Skipped 1 previous similar message [ 9096.203442] Lustre: Skipped 1 previous similar message [ 9100.263599] Lustre: server umount lustre-OST0000 complete [ 9106.403339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9106.424730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9106.430914] Lustre: Skipped 1 previous similar message [ 9110.721634] Lustre: server umount lustre-MDT0000 complete [ 9111.521447] LustreError: 290300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9111.535106] LustreError: 290300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9114.390521] LustreError: 291034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980518 with bad export cookie 10746483154107638653 [ 9114.393985] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9114.419602] LustreError: 291034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9114.748576] Lustre: server umount lustre-MDT0001 complete [ 9129.019366] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9132.079816] Key type lgssc unregistered [ 9132.452823] LNet: 293835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9132.472589] LNetError: 293835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9132.500293] LNet: Removed LNI 192.168.202.110@tcp [ 9133.451125] Key type .llcrypt unregistered [ 9133.457396] Key type ._llcrypt unregistered [ 9161.780240] Key type ._llcrypt registered [ 9161.784177] Key type .llcrypt registered [ 9161.905441] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 9178.372889] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9178.902795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9178.944547] alg: No test for adler32 (adler32-zlib) [ 9180.166520] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 9180.407100] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9182.127810] Key type lgssc registered [ 9183.198690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9229.388275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9229.455788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9231.011129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9231.078072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9231.226118] Lustre: lustre-MDT0000: new disk, initializing [ 9231.347829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9231.371914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9236.788812] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9247.567955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9247.746370] Lustre: 297676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9247.846315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9247.856247] Lustre: Skipped 1 previous similar message [ 9247.987248] Lustre: lustre-MDT0001: new disk, initializing [ 9248.142816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9248.200598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9248.213262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9253.674475] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9263.015995] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9268.806717] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9275.662162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9275.999406] Lustre: lustre-OST2710: new disk, initializing [ 9276.003338] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9276.008382] Lustre: 299649:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9276.104215] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9283.242909] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9286.238506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9286.247777] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9286.429057] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9292.080857] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9298.763906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9298.914553] Lustre: lustre-OST03e8: new disk, initializing [ 9298.917792] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9298.923104] Lustre: 300712:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9299.033876] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9306.132057] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9306.730282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9306.748411] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9306.835543] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9313.611984] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9320.635763] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9320.798806] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9326.003181] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9326.238367] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9331.315868] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9331.581270] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9336.615563] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9336.936722] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9342.955978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.962093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9342.978826] Lustre: Skipped 2 previous similar messages [ 9342.993593] Lustre: Skipped 3 previous similar messages [ 9343.696154] Lustre: server umount lustre-MDT0000 complete [ 9347.034167] LustreError: 297671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980751 with bad export cookie 16066792445615035753 [ 9347.052199] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9347.053126] LustreError: 297671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9347.436333] Lustre: server umount lustre-MDT0001 complete [ 9360.197544] Lustre: server umount lustre-OST2710 complete [ 9374.398303] Lustre: server umount lustre-OST03e8 complete [ 9392.222294] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 9402.991167] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9441.020180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.284913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9441.312418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9441.398284] Lustre: lustre-MDT0000: new disk, initializing [ 9441.480553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9441.508563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9446.722314] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9457.157609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9457.351226] Lustre: 305964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9457.420187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9457.423756] Lustre: Skipped 1 previous similar message [ 9457.484952] Lustre: lustre-MDT0001: new disk, initializing [ 9457.600960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9457.649484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9457.658491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9462.001598] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9470.601212] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9475.508149] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9481.364871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9481.643267] Lustre: lustre-OST0000: new disk, initializing [ 9481.646556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9481.651755] Lustre: 307929:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9481.723628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9487.612332] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9490.518989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9490.533500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9490.645143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9497.870984] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9503.076662] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9503.399808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9507.777179] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9507.943294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9511.413569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9511.415448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9511.420810] LustreError: Skipped 1 previous similar message [ 9511.426678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9511.450544] Lustre: Skipped 2 previous similar messages [ 9511.464974] Lustre: Skipped 1 previous similar message [ 9516.076275] Lustre: server umount lustre-OST0000 complete [ 9521.634696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9521.648637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9521.648643] Lustre: Skipped 1 previous similar message [ 9526.759020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9526.771040] Lustre: Skipped 2 previous similar messages [ 9527.220201] Lustre: server umount lustre-MDT0000 complete [ 9531.568679] LustreError: 307940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980935 with bad export cookie 16066792445615037433 [ 9531.572703] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9531.576508] LustreError: 307940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9532.265492] Lustre: server umount lustre-MDT0001 complete [ 9540.963330] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:42:25 (1790980945) [ 9542.475668] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9543.997449] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:42:27 (1790980947) [ 9548.152395] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9576.837811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9577.192519] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9577.206971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9577.366139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9583.318933] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9593.576668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9593.717392] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9593.752702] Lustre: 311791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9599.698680] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9621.242836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9621.560869] Lustre: Denying initial registration attempt from nid 192.168.202.110@tcp, specified as failover [ 9621.567538] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9621.577316] LustreError: 313242:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9621.582408] LustreError: 313242:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9621.588397] LustreError: 313242:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9621.658950] Lustre: server umount lustre-OST0000 complete [ 9621.665299] LustreError: 313242:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9629.668145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9629.681551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9629.700975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9635.755717] Lustre: server umount lustre-MDT0000 complete [ 9637.348949] LustreError: 311802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9637.368291] LustreError: 311802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9639.856729] LustreError: 311784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981044 with bad export cookie 16066792445615038315 [ 9639.862948] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9639.880346] LustreError: 311784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9640.288174] Lustre: server umount lustre-MDT0001 complete [ 9654.158873] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9657.910558] Key type lgssc unregistered [ 9658.267778] LNet: 314363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9658.278951] LNetError: 314363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9658.307288] LNet: Removed LNI 192.168.202.110@tcp [ 9659.299307] Key type .llcrypt unregistered [ 9659.301509] Key type ._llcrypt unregistered [ 9672.220446] Key type ._llcrypt registered [ 9672.223095] Key type .llcrypt registered [ 9672.328809] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9673.913880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9673.989370] alg: No test for adler32 (adler32-zlib) [ 9675.197898] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 9675.474597] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9677.215214] Key type lgssc registered [ 9678.584865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9728.514195] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9737.931986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9737.956179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9739.221854] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9739.280864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9739.491522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.876066] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9752.547758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9752.704621] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9752.725679] Lustre: 317148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9752.942556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9756.891770] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9763.791041] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9770.078994] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9789.805278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9790.058717] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9790.267872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9797.235054] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9804.650769] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9807.841439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9807.851566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9807.870441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9808.381230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9808.390852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.394837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9813.024510] Lustre: server umount lustre-OST0000 complete [ 9818.597237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9818.612104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9818.614624] Lustre: Skipped 1 previous similar message [ 9822.766184] Lustre: server umount lustre-MDT0000 complete [ 9825.859153] LustreError: 317138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981230 with bad export cookie 8252860726363988311 [ 9825.860163] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9825.865571] LustreError: 317138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9826.124791] Lustre: server umount lustre-MDT0001 complete [ 9838.860242] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9841.748235] Key type lgssc unregistered [ 9842.192065] LNet: 320718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9842.205064] LNetError: 320718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9843.246329] LNet: Removed LNI 192.168.202.110@tcp [ 9844.501642] Key type .llcrypt unregistered [ 9844.503737] Key type ._llcrypt unregistered [ 9873.125464] Key type ._llcrypt registered [ 9873.127748] Key type .llcrypt registered [ 9873.278410] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9874.751890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9874.785622] alg: No test for adler32 (adler32-zlib) [ 9875.950342] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 9876.345991] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9878.143269] Key type lgssc registered [ 9879.732297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9887.641489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9887.651850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.062061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9894.084849] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9899.492073] LustreError: 322151:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9899.507373] LustreError: 322151:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9903.645085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9904.031234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9909.283850] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9918.477654] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9924.955411] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9933.670995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9933.861040] LustreError: 323975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.061964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9939.944311] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9946.784778] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9960.419706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9960.431649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9960.447935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9964.100382] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 9965.034169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9966.575744] Lustre: server umount lustre-MDT0000 complete [ 9969.181182] LustreError: 322885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.837363] LustreError: 322131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981375 with bad export cookie 7748088891993344737 [ 9970.848206] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9970.849569] LustreError: 322131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9974.316875] LustreError: 322146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9974.343250] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [ 9974.347882] LustreError: 322146:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9974.375788] Lustre: Skipped 3 previous similar messages [ 9975.266686] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.272334] Lustre: Skipped 1 previous similar message [ 9977.227569] Lustre: server umount lustre-MDT0001 complete [ 9982.958184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9992.028457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9996.708141] LustreError: 321452:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002c9c33d2 x1877979986703360/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9996.953465] LustreError: 323985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9996.962621] LustreError: 323985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9997.000177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.045402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10003.975968] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10005.101270] LustreError: 323986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10005.103844] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10005.117247] LustreError: 323986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10013.673649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.196292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10015.209629] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10019.236121] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10027.912769] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10030.239665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10036.894742] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10038.555278] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10044.898018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10044.903056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.929057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10044.936162] Lustre: Skipped 1 previous similar message [10050.384717] Lustre: server umount lustre-OST0000 complete [10055.154748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10055.166349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10055.180275] Lustre: Skipped 1 previous similar message [10055.189603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10055.192141] Lustre: Skipped 2 previous similar messages [10060.853646] Lustre: server umount lustre-MDT0000 complete [10061.283304] LustreError: 325765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10061.291372] LustreError: 325765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10064.920555] LustreError: 322133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981469 with bad export cookie 7748088891993352577 [10064.931825] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10064.932649] LustreError: 322133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10065.362421] Lustre: server umount lustre-MDT0001 complete [10079.536970] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [10082.441883] Key type lgssc unregistered [10082.801815] LNet: 328439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10082.823584] LNetError: 328439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10083.877555] LNet: Removed LNI 192.168.202.110@tcp [10084.784630] Key type .llcrypt unregistered [10084.786928] Key type ._llcrypt unregistered [10114.485878] Key type ._llcrypt registered [10114.488724] Key type .llcrypt registered [10114.689606] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10116.253912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10116.308500] alg: No test for adler32 (adler32-zlib) [10117.474280] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [10117.657851] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [10119.377254] Key type lgssc registered [10121.245031] Lustre: Echo OBD driver; http://www.lustre.org/ [10129.908139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10129.976574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10131.716546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10136.232089] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10141.167454] LustreError: 329874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.203141] LustreError: 329874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10145.004967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10145.410913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10150.147902] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10158.070809] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10164.360547] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10169.314739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10169.327838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.337883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.349748] Lustre: Skipped 2 previous similar messages [10175.462631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10175.476348] Lustre: Skipped 1 previous similar message [10180.577022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10180.580480] Lustre: Skipped 1 previous similar message [10182.113971] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10182.345734] Lustre: server umount lustre-MDT0000 complete [10185.696661] LustreError: 329871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10185.709540] LustreError: 329871:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10186.604869] LustreError: 329856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981590 with bad export cookie 16662341433205158247 [10186.609147] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10186.611591] LustreError: 329856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10186.902368] Lustre: server umount lustre-MDT0001 complete [10201.513605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10201.840103] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10201.910517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10202.134687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10207.494197] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10216.737212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.876183] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10216.897864] Lustre: 332466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10216.983945] LustreError: 332477:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10217.192305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10221.783056] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10229.144942] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10235.351988] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10245.486982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10245.762212] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10245.779501] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10245.796832] LustreError: 334349:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10245.805886] LustreError: 334349:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10245.811210] LustreError: 334349:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10245.875070] Lustre: server umount lustre-OST0000 complete [10245.883490] LustreError: 334349:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10255.722740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10255.861916] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10256.013441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10262.784596] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10264.800945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10269.655284] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10277.209407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10277.331493] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10277.440633] Lustre: lustre-OST0001: new disk, initializing [10277.446284] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10277.450760] Lustre: 335893:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10277.564332] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10282.953382] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10286.723043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10286.746620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10286.912381] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10291.447040] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10297.319209] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10297.323784] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10297.333173] LustreError: Skipped 1 previous similar message [10297.334997] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10297.369417] Lustre: Skipped 2 previous similar messages [10300.713467] Lustre: server umount lustre-OST0001 complete [10302.441912] LustreError: 334814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.472819] LustreError: 334814:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10306.016793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10306.022320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10306.027863] LustreError: Skipped 1 previous similar message [10306.031499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10306.031508] Lustre: Skipped 1 previous similar message [10306.077650] Lustre: Skipped 1 previous similar message [10307.575700] LustreError: 334815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10307.595108] LustreError: 334815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10311.945302] Lustre: server umount lustre-OST0000 complete [10317.792797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10322.919305] LustreError: 333225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10323.063915] Lustre: server umount lustre-MDT0000 complete [10326.596577] LustreError: 334834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981730 with bad export cookie 16662341433205158807 [10326.598138] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10326.621162] LustreError: 334834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10327.211646] Lustre: server umount lustre-MDT0001 complete [10337.863877] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [10340.523227] Key type lgssc unregistered [10340.827636] LNet: 337850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10340.834842] LNetError: 337850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10341.882445] LNet: Removed LNI 192.168.202.110@tcp [10342.825062] Key type .llcrypt unregistered [10342.826131] Key type ._llcrypt unregistered [10416.449476] Key type ._llcrypt registered [10416.453618] Key type .llcrypt registered [10416.627489] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [10432.227464] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10433.718574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10433.730461] alg: No test for adler32 (adler32-zlib) [10434.694185] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [10435.044771] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [10436.720587] Key type lgssc registered [10438.357049] Lustre: Echo OBD driver; http://www.lustre.org/ [10478.709851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10478.786760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10480.117508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10480.188000] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10480.322529] Lustre: lustre-MDT0000: new disk, initializing [10480.437097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10480.469926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10485.667606] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10496.665781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10496.786918] Lustre: 342801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10496.879970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10496.887621] Lustre: Skipped 1 previous similar message [10497.026585] Lustre: lustre-MDT0001: new disk, initializing [10497.151585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10497.201298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10497.218929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10502.500786] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10513.348932] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10519.614080] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10527.384800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10527.730414] Lustre: lustre-OST0000: new disk, initializing [10527.734213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10527.740196] Lustre: 344771:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10527.850275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10535.010668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10535.028620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10535.191745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10535.808821] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10546.276657] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10551.135854] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10551.416557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10555.755168] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10555.952453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10560.993473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10560.996582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10561.006179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10561.006671] Lustre: Skipped 1 previous similar message [10561.022854] Lustre: Skipped 1 previous similar message [10566.131777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10571.241725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10571.248867] Lustre: Skipped 1 previous similar message [10572.255277] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10572.569022] Lustre: server umount lustre-OST0000 complete [10578.912795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10578.919346] LustreError: Skipped 1 previous similar message [10578.930663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.958326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10578.965677] Lustre: Skipped 1 previous similar message [10581.476279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10586.594296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10586.609854] Lustre: Skipped 3 previous similar messages [10591.199665] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10591.417095] Lustre: server umount lustre-MDT0000 complete [10594.814832] LustreError: 342792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981998 with bad export cookie 14455434170478418314 [10594.823827] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10594.829606] LustreError: 342792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10595.060897] Lustre: server umount lustre-MDT0001 complete [10604.273326] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:00:07 (1790982007) [10611.458602] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:00:15 (1790982015) [10619.699180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.099663] LustreError: 347528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.118460] LustreError: 347528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10620.184629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10625.191408] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10625.512775] LustreError: 347529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10630.628854] LustreError: 347528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.912110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10634.303934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10639.307641] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10647.565750] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10654.213754] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10661.654029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10662.031186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10668.247855] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10675.280285] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10687.263586] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10688.417719] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10689.887358] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10691.715916] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10693.890396] Lustre: Failing over lustre-MDT0000 [10694.465534] Lustre: server umount lustre-MDT0000 complete [10695.148734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10695.151649] LustreError: 347525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10695.172126] Lustre: Skipped 1 previous similar message [10695.191688] LustreError: 347525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10700.258157] LustreError: 347529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.267585] LustreError: 347529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10708.492364] LustreError: 348909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10708.510408] LustreError: 348909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10711.328310] Lustre: 341015:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790982099/real 1790982099] req@00000000335d3e88 x1877980572886528/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790982115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10711.349822] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10712.572743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10721.962549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10722.014609] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10723.882448] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10727.409132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10727.464343] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10727.543226] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10727.547658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10737.665689] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10739.400894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10743.362161] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10745.462429] Lustre: Failing over lustre-MDT0000 [10745.912669] Lustre: server umount lustre-MDT0000 complete [10747.873068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10747.892034] Lustre: Skipped 3 previous similar messages [10747.899470] LustreError: 347525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10747.932227] LustreError: 347525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [10764.129488] Lustre: 341013:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790982152/real 1790982152] req@0000000062c3b0c5 x1877980572907392/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790982168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10764.152804] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10765.000688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10774.532755] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc89c04ca3568e41f [10774.550970] Lustre: MGC192.168.202.110@tcp: Connection restored to 0@lo (at 0@lo) [10774.563628] Lustre: Skipped 2 previous similar messages [10774.762912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10774.827802] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10775.052954] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10780.173610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10780.193143] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10780.216290] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10780.283907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10789.476978] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10790.951864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10800.618926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10800.633951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10800.637870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10800.653027] Lustre: Skipped 1 previous similar message [10800.674748] Lustre: Skipped 2 previous similar messages [10803.155889] Lustre: server umount lustre-MDT0000 complete [10805.734175] LustreError: 347524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10805.753942] LustreError: 347524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [10807.495299] LustreError: 347510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982211 with bad export cookie 14455434170478421023 [10807.501856] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10807.507956] LustreError: 347510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10808.113709] Lustre: server umount lustre-MDT0001 complete [10822.527896] Lustre: server umount lustre-OST0000 complete [10837.245843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10841.129756] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10849.349632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.675970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10849.681559] Lustre: Skipped 1 previous similar message [10854.250249] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10861.409883] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10868.122580] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10875.736674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10882.232395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10883.334273] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10891.066805] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10903.010357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10903.015397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10903.020709] LustreError: Skipped 1 previous similar message [10903.039533] Lustre: Skipped 1 previous similar message [10903.056428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10906.710449] Lustre: server umount lustre-OST0000 complete [10911.201448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10911.211431] LustreError: Skipped 1 previous similar message [10916.757903] Lustre: server umount lustre-MDT0000 complete [10919.987509] LustreError: 353888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982324 with bad export cookie 14455434170478421779 [10920.005710] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10920.349752] Lustre: server umount lustre-MDT0001 complete [10932.101919] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [10935.122303] Key type lgssc unregistered [10935.382739] LNet: 357433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10935.387788] LNetError: 357433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10935.408898] LNet: Removed LNI 192.168.202.110@tcp [10936.210087] Key type .llcrypt unregistered [10936.211307] Key type ._llcrypt unregistered [10960.658477] Key type ._llcrypt registered [10960.659919] Key type .llcrypt registered [10960.757692] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10961.994301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10962.018165] alg: No test for adler32 (adler32-zlib) [10963.091066] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [10963.352725] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [10965.063194] Key type lgssc registered [10966.186889] Lustre: Echo OBD driver; http://www.lustre.org/ [10972.995319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10973.030284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10974.394113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10978.163504] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10983.908288] LustreError: 358959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10983.926423] LustreError: 358959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10986.398413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10986.725908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10990.757601] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10997.446819] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11003.332805] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11010.386514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11010.840588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11017.061068] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11019.140790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11024.151490] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11037.716489] Lustre: server umount lustre-MDT0000 complete [11038.188433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11038.201832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11038.215550] LustreError: 358959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.713433] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11039.716954] LustreError: 358958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.757227] LustreError: 358958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11041.185980] LustreError: 359692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982445 with bad export cookie 7257834903696534555 [11041.198852] LustreError: 359692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11041.204513] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11041.479716] Lustre: server umount lustre-MDT0001 complete [11055.854599] Lustre: server umount lustre-OST0000 complete [11068.657963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11069.147393] LustreError: 362669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11069.228955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11073.403499] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11074.531050] LustreError: 362670:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11082.612703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.955463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11087.697418] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11096.179920] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11101.064591] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11108.333924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11108.651752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11114.771903] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11114.809657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11120.907535] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11134.433218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11134.445742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11134.474336] Lustre: Skipped 1 previous similar message [11134.489353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11135.460397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11135.465892] Lustre: Skipped 2 previous similar messages [11137.804784] Lustre: server umount lustre-MDT0000 complete [11140.577502] LustreError: 362666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11140.603727] LustreError: 362666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11141.332348] LustreError: 362651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982545 with bad export cookie 7257834903696535675 [11141.333968] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11141.340805] LustreError: 362651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11141.570311] Lustre: server umount lustre-MDT0001 complete [11155.785426] Lustre: server umount lustre-OST0000 complete [11171.882269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11172.546168] LustreError: 366578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11172.655433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11178.877457] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11187.068976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.370657] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11202.381270] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11208.059689] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11215.962089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11216.507543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11216.514971] Lustre: Skipped 1 previous similar message [11217.537303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11221.797453] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11229.737889] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11237.344346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11237.350515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.361552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11237.382936] Lustre: Skipped 3 previous similar messages [11237.403423] Lustre: Skipped 1 previous similar message [11240.013665] Lustre: server umount lustre-OST0000 complete [11244.001031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11244.022454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11244.036358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11252.704496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11252.711568] Lustre: Skipped 3 previous similar messages [11258.335140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11258.701871] Lustre: server umount lustre-MDT0000 complete [11262.603830] LustreError: 368423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982666 with bad export cookie 7257834903696536697 [11262.629837] LustreError: 368423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11262.644518] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11263.119700] Lustre: server umount lustre-MDT0001 complete [11275.023550] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11278.428640] Key type lgssc unregistered [11278.789838] LNet: 370113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11278.806844] LNetError: 370113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11278.847914] LNet: Removed LNI 192.168.202.110@tcp [11279.989217] Key type .llcrypt unregistered [11279.996300] Key type ._llcrypt unregistered [11307.461641] Key type ._llcrypt registered [11307.464131] Key type .llcrypt registered [11307.597828] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11309.094294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11309.127462] alg: No test for adler32 (adler32-zlib) [11310.395302] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [11310.894293] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11312.671741] Key type lgssc registered [11314.176773] Lustre: Echo OBD driver; http://www.lustre.org/ [11323.896935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11323.924414] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11323.930811] LustreError: 371583:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11323.947889] LustreError: 371583:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11323.956130] LustreError: 371583:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11323.960715] LustreError: 371583:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11323.978348] LustreError: 371583:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11333.292776] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11333.308970] LustreError: 371982:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11333.320739] LustreError: 371982:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11333.326820] LustreError: 371982:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11333.338988] LustreError: 371982:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11333.356768] LustreError: 371982:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11353.970835] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11357.476858] Key type lgssc unregistered [11357.714770] LNet: 372985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11357.719802] LNetError: 372985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11357.730925] LNet: Removed LNI 192.168.202.110@tcp [11359.119793] Key type .llcrypt unregistered [11359.121594] Key type ._llcrypt unregistered [11376.788455] Key type ._llcrypt registered [11376.791256] Key type .llcrypt registered [11376.887210] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [11393.360775] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11394.354095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11394.425047] alg: No test for adler32 (adler32-zlib) [11395.571798] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [11395.917252] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11397.703719] Key type lgssc registered [11398.997430] Lustre: Echo OBD driver; http://www.lustre.org/ [11433.583374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11433.624206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11434.898614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11434.939567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11435.005551] Lustre: lustre-MDT0000: new disk, initializing [11435.146056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11435.166657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11439.482705] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11452.988118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.197120] Lustre: 376160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11453.302289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11453.312133] Lustre: Skipped 1 previous similar message [11453.479551] Lustre: lustre-MDT0001: new disk, initializing [11453.720047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11453.805590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11453.829535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11460.524853] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11472.244772] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11478.333415] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11486.281984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11486.620441] Lustre: lustre-OST0000: new disk, initializing [11486.630660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11486.639982] Lustre: 378129:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11486.774813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11494.091201] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11495.051846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11495.069792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11495.180444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11503.757665] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11508.755406] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11509.114615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11514.556182] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11514.891840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11520.994680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11520.995658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11521.046894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11521.076980] Lustre: Skipped 1 previous similar message [11523.213923] Lustre: server umount lustre-OST0000 complete [11530.721549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11530.728027] LustreError: Skipped 1 previous similar message [11530.732951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.749558] Lustre: Skipped 1 previous similar message [11530.759787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11534.172347] Lustre: server umount lustre-MDT0000 complete [11538.525432] LustreError: 376151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982942 with bad export cookie 14887457132956448808 [11538.544309] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11538.549084] LustreError: 376151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11539.549782] Lustre: server umount lustre-MDT0001 complete [11549.855775] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11551.815945] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:15:55 (1790982955) [11559.754667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11560.344029] LustreError: 380589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11560.370512] LustreError: 380589:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11560.466638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11565.538119] LustreError: 380590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11565.641136] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11570.656183] LustreError: 380589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11574.646486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11574.996768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11579.849198] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11587.716428] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11594.274341] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11601.553439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11602.028287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11609.728736] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11618.051819] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11625.303562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11625.420959] Lustre: lustre-OST0001: new disk, initializing [11625.424121] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11625.428905] Lustre: 383426:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11625.513398] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11633.800176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11633.814156] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11633.962332] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11633.985538] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11645.622369] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11652.675117] Lustre: lustre-OST0001: Not available for connect from 192.168.202.10@tcp (stopping) [11652.691204] Lustre: Skipped 2 previous similar messages [11654.625805] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11654.642606] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11654.654245] Lustre: Skipped 1 previous similar message [11656.060427] Lustre: server umount lustre-OST0001 complete [11657.748447] LustreError: 383784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11664.878777] LustreError: 383784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11664.901680] LustreError: 383784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11675.112981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11675.132986] Lustre: Skipped 1 previous similar message [11675.142282] LustreError: 382412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.145074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11675.170583] LustreError: 382412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11675.197873] Lustre: Skipped 3 previous similar messages [11679.778853] Lustre: server umount lustre-OST0000 complete [11685.345086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11685.348323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11685.355150] Lustre: Skipped 2 previous similar messages [11685.370301] Lustre: Skipped 1 previous similar message [11689.879992] Lustre: server umount lustre-MDT0000 complete [11694.193484] LustreError: 380569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983098 with bad export cookie 14887457132956449690 [11694.210167] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11694.214693] LustreError: 380569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11700.831664] Lustre: server umount lustre-MDT0001 complete [11715.982882] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11719.024779] Key type lgssc unregistered [11719.222684] LNet: 385418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11719.226717] LNetError: 385418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11719.242236] LNet: Removed LNI 192.168.202.110@tcp [11719.832247] Key type .llcrypt unregistered [11719.834945] Key type ._llcrypt unregistered [11773.669156] Key type ._llcrypt registered [11773.670771] Key type .llcrypt registered [11773.770805] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11775.156962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11775.253776] alg: No test for adler32 (adler32-zlib) [11776.378930] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [11776.675281] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11778.463227] Key type lgssc registered [11779.857679] Lustre: Echo OBD driver; http://www.lustre.org/ [11788.025541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11788.036919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.393907] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11789.418231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11789.679448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11794.506913] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11802.757501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11802.889450] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11802.919777] Lustre: 388230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11803.259343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11808.192808] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11816.598764] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11822.254093] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11827.173903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11827.176207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11827.193415] Lustre: Skipped 1 previous similar message [11827.211852] Lustre: Skipped 1 previous similar message [11830.586996] Lustre: server umount lustre-MDT0000 complete [11834.421450] LustreError: 388223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983238 with bad export cookie 11964854523124947154 [11834.433505] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11834.437452] LustreError: 388223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11834.983430] Lustre: server umount lustre-MDT0001 complete [11845.474920] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:20:49 (1790983249) [11853.054584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.797805] LustreError: 390745:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.836169] LustreError: 390745:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11853.931724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11858.916467] LustreError: 390746:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.878710] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11864.037518] LustreError: 390745:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11869.153979] LustreError: 390746:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11869.722928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11870.140638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11875.998623] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11883.908407] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11889.758887] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11897.667426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11897.907129] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11898.142920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11904.939369] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11912.552982] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11921.009629] Lustre: Permanently deactivating lustre-OST0000 [11921.023234] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11926.552239] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11929.249525] LustreError: 393683:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11929.255325] LustreError: 393683:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11929.259963] LustreError: 393683:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11932.713640] LustreError: 393732:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11932.722628] LustreError: 393732:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11937.034559] Lustre: server umount lustre-OST0000 complete [11940.192994] LustreError: 393981:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11940.197610] LustreError: 393981:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11946.465143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11946.470572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11946.483525] Lustre: Skipped 1 previous similar message [11946.502292] Lustre: Skipped 1 previous similar message [11951.584160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11951.589230] Lustre: Skipped 1 previous similar message [11952.800378] Lustre: server umount lustre-MDT0000 complete [11956.484436] LustreError: 390727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983360 with bad export cookie 11964854523124947735 [11956.497503] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11956.502320] LustreError: 390727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11966.333928] Lustre: server umount lustre-MDT0001 complete [11975.408322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.589795] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11983.444737] LustreError: 395142:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11989.925348] Lustre: 395480:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.110@tcp [11992.213855] Lustre: server umount MGS complete [12004.026800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12004.558082] LustreError: 396204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.597588] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12004.607934] Lustre: Skipped 1 previous similar message [12004.715327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12010.380371] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12015.080728] LustreError: 396205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12015.105890] LustreError: 396205:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12019.620265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12019.852206] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12024.946612] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12032.962661] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12039.003177] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12041.181396] Lustre: Permanently reactivating lustre-OST0000 [12041.208155] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12041.219415] Lustre: Skipped 2 previous similar messages [12068.831795] LustreError: 397984:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12076.093526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12076.384877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12076.388428] Lustre: Skipped 1 previous similar message [12083.308601] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12090.534558] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12094.947074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12094.968728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12094.979653] Lustre: Skipped 1 previous similar message [12100.064511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12100.088138] Lustre: Skipped 1 previous similar message [12100.265722] Lustre: server umount lustre-OST0000 complete [12105.205853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12105.230245] Lustre: Skipped 1 previous similar message [12110.005510] Lustre: server umount lustre-MDT0000 complete [12110.305763] LustreError: 396204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12110.323398] LustreError: 396204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12114.310683] LustreError: 396186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983518 with bad export cookie 11964854523124948785 [12114.324236] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12114.327173] LustreError: 396186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12114.728922] Lustre: server umount lustre-MDT0001 complete [12126.614101] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12129.842385] Key type lgssc unregistered [12130.116799] LNet: 400567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12130.122804] LNetError: 400567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12131.191091] LNet: Removed LNI 192.168.202.110@tcp [12132.196298] Key type .llcrypt unregistered [12132.199806] Key type ._llcrypt unregistered [12151.572533] Key type ._llcrypt registered [12151.575381] Key type .llcrypt registered [12151.673613] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [12169.553313] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12171.085749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12171.094425] alg: No test for adler32 (adler32-zlib) [12172.235313] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [12172.477219] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12174.143319] Key type lgssc registered [12175.644830] Lustre: Echo OBD driver; http://www.lustre.org/ [12211.603507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12211.632552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12212.909777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12212.948870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12213.067197] Lustre: lustre-MDT0000: new disk, initializing [12213.150728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12213.167560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12218.141946] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12228.591811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12228.687647] Lustre: 403741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12228.728847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12228.732475] Lustre: Skipped 1 previous similar message [12228.816775] Lustre: lustre-MDT0001: new disk, initializing [12228.934653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12228.974140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12228.988185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12233.513687] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12242.956924] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12248.420493] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12255.447924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12255.694177] Lustre: lustre-OST0000: new disk, initializing [12255.698614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12255.708598] Lustre: 405709:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12255.798237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12262.058115] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12263.980385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12263.986566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12264.088360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12272.263836] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12277.532903] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12277.826395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12283.026824] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12283.336123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12290.019237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12290.020095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12290.029250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12290.042147] Lustre: Skipped 1 previous similar message [12290.062667] Lustre: Skipped 1 previous similar message [12291.604984] Lustre: server umount lustre-OST0000 complete [12295.648054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12295.682984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12295.707390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12300.260962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12300.261606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12300.277442] Lustre: Skipped 1 previous similar message [12302.183675] Lustre: server umount lustre-MDT0000 complete [12305.839752] LustreError: 404537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983710 with bad export cookie 14930795444524357993 [12305.844073] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12305.849277] LustreError: 404537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12306.231699] Lustre: server umount lustre-MDT0001 complete [12314.565653] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12316.154512] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:28:40 (1790983720) [12323.654213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12324.028624] LustreError: 408169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12324.042843] LustreError: 408169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12324.101760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12328.701631] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12329.441863] LustreError: 408170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12334.567831] LustreError: 408169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12337.553035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12338.084306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12342.859037] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12351.269684] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12357.956579] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12365.845522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12366.405783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12373.363658] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12381.623784] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12383.782199] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12386.552684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12394.469150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12394.483185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12398.642673] Lustre: server umount lustre-OST0000 complete [12404.704052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12404.708147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12404.712088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12404.712096] Lustre: Skipped 1 previous similar message [12404.744977] Lustre: Skipped 2 previous similar messages [12409.642813] Lustre: server umount lustre-MDT0000 complete [12409.826238] LustreError: 410764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12409.833698] LustreError: 410764:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12413.091301] LustreError: 408149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983817 with bad export cookie 14930795444524358875 [12413.104713] LustreError: 408149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12413.110342] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12413.452938] Lustre: server umount lustre-MDT0001 complete [12425.168976] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12428.418596] Key type lgssc unregistered [12428.817522] LNet: 411895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12428.853942] LNetError: 411895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12428.879735] LNet: Removed LNI 192.168.202.110@tcp [12429.761842] Key type .llcrypt unregistered [12429.764724] Key type ._llcrypt unregistered [12455.319195] Key type ._llcrypt registered [12455.322921] Key type .llcrypt registered [12455.552920] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12455.786421] Key type .llcrypt unregistered [12455.789393] Key type ._llcrypt unregistered [12479.463947] Key type ._llcrypt registered [12479.469557] Key type .llcrypt registered [12479.567396] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12481.198096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12481.419075] alg: No test for adler32 (adler32-zlib) [12482.650768] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [12482.968230] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12484.719226] Key type lgssc registered [12486.648202] Lustre: Echo OBD driver; http://www.lustre.org/ [12496.825987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12496.841348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12498.190147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12504.640654] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12508.640778] LustreError: 414124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12508.654439] LustreError: 414124:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12509.673236] LustreError: 414125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12514.034926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12514.322167] LustreError: 414124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12514.338898] LustreError: 414124:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12514.345643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12519.403833] LustreError: 414125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12519.429668] LustreError: 414125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12522.794697] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12524.515256] LustreError: 414119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12524.528941] LustreError: 414119:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12524.536059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12532.250650] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12534.754712] LustreError: 414124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12534.768782] LustreError: 414124:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12540.246267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12540.662637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12545.377589] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12555.233966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12555.251130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12555.268926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12556.258383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12556.264038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12556.267483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12556.311133] Lustre: Skipped 1 previous similar message [12560.405333] Lustre: server umount lustre-OST0000 complete [12566.496477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12566.500142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12566.515292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12566.522888] Lustre: Skipped 1 previous similar message [12570.668535] Lustre: server umount lustre-MDT0000 complete [12571.616343] LustreError: 414125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12571.630707] LustreError: 414125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12574.295625] LustreError: 414104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983978 with bad export cookie 8882989243636843378 [12574.310308] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12574.672961] Lustre: server umount lustre-MDT0001 complete [12586.736508] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12589.752705] Key type lgssc unregistered [12590.125736] LNet: 417275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12590.133368] LNetError: 417275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12590.150167] LNet: Removed LNI 192.168.202.110@tcp [12591.002088] Key type .llcrypt unregistered [12591.008074] Key type ._llcrypt unregistered [12617.313808] Key type ._llcrypt registered [12617.315238] Key type .llcrypt registered [12617.409967] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12618.784428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12618.844597] alg: No test for adler32 (adler32-zlib) [12619.997167] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [12620.241289] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12621.959243] Key type lgssc registered [12623.264183] Lustre: Echo OBD driver; http://www.lustre.org/ [12632.220101] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12632.252492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12647.775642] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12652.900479] LustreError: 418680:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.110@tcp: failed processing log, type 4: rc = -110 [12679.519680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12686.635163] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12695.315678] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12703.749764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12704.439371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12704.541238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12709.861188] LustreError: 418704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.880864] LustreError: 418704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12710.011692] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12714.033326] LustreError: 419715:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12714.069141] LustreError: 419715:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12719.087130] LustreError: 419715:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12719.123547] LustreError: 419715:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12720.462661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12721.000682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12726.207580] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12733.739848] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12740.198431] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12749.004889] Lustre: server umount lustre-OST0000 complete [12753.048702] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12753.054670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.080264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12756.961578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12756.962435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12762.086211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12762.095874] Lustre: Skipped 2 previous similar messages [12763.335524] Lustre: server umount lustre-MDT0000 complete [12767.208742] LustreError: 420706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12767.233423] LustreError: 420706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12767.351033] LustreError: 418686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984171 with bad export cookie 4237320648004049760 [12767.368876] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12777.248758] Lustre: server umount lustre-MDT0001 complete [12789.265265] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12792.377866] Key type lgssc unregistered [12792.756518] LNet: 422249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12792.761075] LNetError: 422249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12792.781416] LNet: Removed LNI 192.168.202.110@tcp [12793.685621] Key type .llcrypt unregistered [12793.687403] Key type ._llcrypt unregistered [12820.184061] Key type ._llcrypt registered [12820.185789] Key type .llcrypt registered [12820.290245] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12821.329464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12821.382614] alg: No test for adler32 (adler32-zlib) [12822.598737] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [12822.916921] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12824.657739] Key type lgssc registered [12826.054129] Lustre: Echo OBD driver; http://www.lustre.org/ [12833.817693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12833.841115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12835.406773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12841.234856] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12845.540939] LustreError: 423681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12845.555559] LustreError: 423681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12850.515505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12850.671280] LustreError: 423682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.947791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12856.227029] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12865.434713] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12872.109408] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12880.247692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12880.700314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12881.762076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12888.027793] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12896.116414] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12901.368652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12901.373755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12901.397071] Lustre: Skipped 1 previous similar message [12901.420288] Lustre: Skipped 2 previous similar messages [12902.959725] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [12906.484601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12906.495756] Lustre: Skipped 1 previous similar message [12906.506561] LustreError: 424669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12906.613657] Lustre: server umount lustre-MDT0000 complete [12909.655517] LustreError: 423662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.10@tcp arrived at 1790984313 with bad export cookie 3530418879730839134 [12909.692326] LustreError: 423662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12911.587784] LustreError: 423682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12911.620279] LustreError: 423682:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12912.513206] LustreError: 424417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984316 with bad export cookie 3530418879730838238 [12912.541049] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12912.541474] LustreError: 424417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12916.706995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12916.709848] LustreError: 423681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12916.719272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.719283] Lustre: Skipped 1 previous similar message [12916.726212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12916.799287] LustreError: 423681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12918.914978] Lustre: server umount lustre-OST0000 complete [12926.945166] LustreError: 424669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12926.954608] LustreError: 424669:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12928.689318] Lustre: server umount lustre-MDT0001 complete [12943.118029] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12946.194962] Key type lgssc unregistered [12946.499902] LNet: 427368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12946.508402] LNetError: 427368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12946.540717] LNet: Removed LNI 192.168.202.110@tcp [12947.896778] Key type .llcrypt unregistered [12947.899875] Key type ._llcrypt unregistered [12978.490423] Key type ._llcrypt registered [12978.491919] Key type .llcrypt registered [12978.645463] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12980.458541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12980.540380] alg: No test for adler32 (adler32-zlib) [12981.605615] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [12981.900576] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12983.626157] Key type lgssc registered [12985.123391] Lustre: Echo OBD driver; http://www.lustre.org/ [12994.850676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12994.883369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12996.422589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13001.608828] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13006.823419] LustreError: 428800:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13006.846974] LustreError: 428800:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13011.937012] LustreError: 428801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13012.749628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13013.249197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13018.144562] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13026.921338] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13034.578383] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13042.721097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13042.974812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13050.248343] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13059.210871] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13066.584606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13074.920061] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13074.933246] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13074.960193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13076.523690] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [13076.967068] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13081.615931] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [13081.618228] Lustre: Skipped 2 previous similar messages [13086.739607] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [13086.747167] Lustre: Skipped 2 previous similar messages [13089.247468] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13089.511371] Lustre: server umount lustre-MDT0001 complete [13091.894305] LustreError: 428797:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13097.440835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13097.442565] LustreError: 429528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13097.443240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13097.446295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13097.506263] LustreError: 429528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13097.531379] Lustre: Skipped 2 previous similar messages [13102.562910] LustreError: 428800:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13103.408566] Lustre: server umount lustre-OST0000 complete [13109.420944] Lustre: server umount lustre-MDT0000 complete [13126.073087] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13129.092624] Key type lgssc unregistered [13129.402154] LNet: 432508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13129.405495] LNetError: 432508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13129.440220] LNet: Removed LNI 192.168.202.110@tcp [13130.652808] Key type .llcrypt unregistered [13130.658940] Key type ._llcrypt unregistered [13160.223690] Key type ._llcrypt registered [13160.225953] Key type .llcrypt registered [13160.344237] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [13175.619402] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13177.091258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13177.136601] alg: No test for adler32 (adler32-zlib) [13178.467557] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [13178.890670] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [13180.626253] Key type lgssc registered [13182.349152] Lustre: Echo OBD driver; http://www.lustre.org/ [13228.101615] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13246.136472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13246.169476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13247.451844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13247.476570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13247.577984] Lustre: lustre-MDT0000: new disk, initializing [13247.683709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13247.709586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13252.066646] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13267.217479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13267.513937] Lustre: lustre-OST0000: new disk, initializing [13267.517782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13267.521438] Lustre: Skipped 1 previous similar message [13267.525948] Lustre: 437559:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13267.592525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13275.371897] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13277.344151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13277.357891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13277.427865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13286.018881] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13302.988341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13303.114474] Lustre: 436542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13303.188342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13303.256789] Lustre: lustre-MDT0001: new disk, initializing [13303.439694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13303.511436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13303.542583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13308.801438] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13318.895453] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13319.282462] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13324.481411] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13324.682236] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13332.964892] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13332.968387] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13332.987287] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13332.999192] Lustre: Skipped 1 previous similar message [13334.050709] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [13336.266321] Lustre: server umount lustre-MDT0001 complete [13343.204423] LustreError: 440130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13343.217430] LustreError: 440130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13348.322382] LustreError: 436548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13348.339135] LustreError: 436548:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13353.444752] LustreError: 436549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13353.460767] LustreError: 436549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13358.563821] LustreError: 436554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13358.564634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13358.600973] Lustre: Skipped 1 previous similar message [13358.607787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13363.695209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13363.697879] LustreError: 436553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13368.801752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13370.336347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13370.580657] Lustre: server umount lustre-OST0000 complete [13373.936792] LustreError: 436553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13373.976792] LustreError: 436553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13382.782798] Lustre: server umount lustre-MDT0000 complete [13400.030649] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13403.397282] Key type lgssc unregistered [13403.679725] LNet: 441312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13403.691644] LNetError: 441312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13403.719718] LNet: Removed LNI 192.168.202.110@tcp [13404.344625] Key type .llcrypt unregistered [13404.346614] Key type ._llcrypt unregistered [13455.801108] Key type ._llcrypt registered [13455.802561] Key type .llcrypt registered [13455.946691] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13456.319577] Key type .llcrypt unregistered [13456.322055] Key type ._llcrypt unregistered [13475.736280] Key type ._llcrypt registered [13475.740708] Key type .llcrypt registered [13475.957119] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13478.071698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13478.129268] alg: No test for adler32 (adler32-zlib) [13479.448248] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [13479.784145] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [13481.583381] Key type lgssc registered [13483.213397] Lustre: Echo OBD driver; http://www.lustre.org/ [13514.726812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13514.795751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13516.116599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13516.137954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13516.197677] Lustre: lustre-MDT0000: new disk, initializing [13516.288816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13516.313224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13521.681565] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13534.289190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13534.511954] Lustre: 444711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13534.591766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13534.596801] Lustre: Skipped 1 previous similar message [13534.741887] Lustre: lustre-MDT0001: new disk, initializing [13534.897895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13534.971815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13534.997313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13540.019210] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13551.966609] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13559.560088] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13567.125414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13567.442291] Lustre: lustre-OST0000: new disk, initializing [13567.453086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13567.466689] Lustre: 446681:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13567.595431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13574.923766] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13577.820347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13577.837496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13577.938232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13585.959251] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13593.573157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13593.584652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13593.588672] Lustre: Skipped 1 previous similar message [13593.606681] Lustre: Skipped 2 previous similar messages [13598.690638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13599.370536] Lustre: server umount lustre-MDT0000 complete [13604.065456] LustreError: 445508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985008 with bad export cookie 2813608969547591396 [13604.069088] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13604.074811] LustreError: 445508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13604.742245] Lustre: server umount lustre-MDT0001 complete [13619.506694] Lustre: server umount lustre-OST0000 complete [13632.161325] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:50:35 (1790985035) [13642.056054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13642.808664] LustreError: 448710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13642.833051] LustreError: 448710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13642.935738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13648.364179] LustreError: 448711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13648.955319] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13653.487293] LustreError: 448710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13658.600139] LustreError: 448711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13659.034932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13659.646355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13665.176419] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13673.280750] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13679.832972] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13701.255875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13701.620646] Lustre: Found index 0 for lustre-OST0000, updating log [13701.935574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13709.052795] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13715.281109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13718.090649] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13724.644113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13724.663180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13724.683411] Lustre: Skipped 1 previous similar message [13724.707431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13724.717236] Lustre: Skipped 2 previous similar messages [13729.965518] Lustre: server umount lustre-OST0000 complete [13746.657357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13746.662239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13746.671038] Lustre: Skipped 1 previous similar message [13746.676108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13746.685413] Lustre: Skipped 3 previous similar messages [13752.801790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13752.811192] Lustre: Skipped 3 previous similar messages [13753.074302] Lustre: server umount lustre-MDT0000 complete [13753.838257] LustreError: 448711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13753.864509] LustreError: 448711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13757.438323] LustreError: 448690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985161 with bad export cookie 2813608969547592782 [13757.449500] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13757.894966] Lustre: server umount lustre-MDT0001 complete [13767.583975] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [13768.367537] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:52:51 (1790985171) [14391.856422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14392.270581] LustreError: 455130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14392.376893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14397.404858] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14397.408367] LustreError: 455131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14402.531581] LustreError: 455130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14407.308169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14407.808382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14412.910363] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14421.339429] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14429.015280] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14446.495146] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [14449.327913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14449.528180] Lustre: Found index 0 for lustre-OST0000, updating log [14449.898471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14457.327442] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14465.320811] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14468.324453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14468.340565] Lustre: Skipped 1 previous similar message [14468.348477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14473.718529] Lustre: server umount lustre-OST0000 complete [14478.441502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14478.474161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14480.546848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14480.553941] Lustre: Skipped 4 previous similar messages [14483.820048] Lustre: server umount lustre-MDT0000 complete [14484.642613] LustreError: 455127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14484.660500] LustreError: 455127:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14488.158893] LustreError: 455113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790985892 with bad export cookie 2813608969547593874 [14488.163689] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14488.175529] LustreError: 455113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14488.489038] Lustre: server umount lustre-MDT0001 complete [14505.887124] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [14522.792519] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [14534.056374] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14570.587117] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:06:14 (1790985974) [14588.118255] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14590.836837] Key type lgssc unregistered [14591.117888] LNet: 461743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14591.126560] LNetError: 461743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14709.926844] LNet: Removed LNI 192.168.202.110@tcp [14710.887551] Key type .llcrypt unregistered [14710.889038] Key type ._llcrypt unregistered [14726.361542] Key type ._llcrypt registered [14726.366578] Key type .llcrypt registered [14726.483968] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14727.839610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14727.898896] alg: No test for adler32 (adler32-zlib) [14728.894353] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [14729.069485] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [14730.711357] Key type lgssc registered [14731.873203] Lustre: Echo OBD driver; http://www.lustre.org/ [14736.120503] LNet: Added LNI 192.168.202.110@tcp100 [8/256/0/180] [14736.881420] LNet: Added LNI 192.168.202.110@tcp101 [8/256/0/180] [14737.705146] LNet: Added LNI 192.168.202.110@tcp102 [8/256/0/180] [14738.467431] LNet: Added LNI 192.168.202.110@tcp103 [8/256/0/180] [14739.351514] LNet: Added LNI 192.168.202.110@tcp104 [8/256/0/180] [14740.143416] LNet: Added LNI 192.168.202.110@tcp105 [8/256/0/180] [14741.131102] LNet: Added LNI 192.168.202.110@tcp106 [8/256/0/180] [14742.062970] LNet: Added LNI 192.168.202.110@tcp107 [8/256/0/180] [14743.056116] LNet: Added LNI 192.168.202.110@tcp108 [8/256/0/180] [14744.042170] LNet: Added LNI 192.168.202.110@tcp109 [8/256/0/180] [14744.840272] LNet: Added LNI 192.168.202.110@tcp110 [8/256/0/180] [14745.649908] LNet: Added LNI 192.168.202.110@tcp111 [8/256/0/180] [14746.595904] LNet: Added LNI 192.168.202.110@tcp112 [8/256/0/180] [14747.584968] LNet: Added LNI 192.168.202.110@tcp113 [8/256/0/180] [14748.447151] LNet: Added LNI 192.168.202.110@tcp114 [8/256/0/180] [14749.356948] LNet: Added LNI 192.168.202.110@tcp115 [8/256/0/180] [14750.320712] LNet: Added LNI 192.168.202.110@tcp116 [8/256/0/180] [14751.257815] LNet: Added LNI 192.168.202.110@tcp117 [8/256/0/180] [14752.332470] LNet: Added LNI 192.168.202.110@tcp118 [8/256/0/180] [14753.301895] LNet: Added LNI 192.168.202.110@tcp119 [8/256/0/180] [14754.219565] LNet: Added LNI 192.168.202.110@tcp120 [8/256/0/180] [14755.150338] LNet: Added LNI 192.168.202.110@tcp121 [8/256/0/180] [14755.977444] LNet: Added LNI 192.168.202.110@tcp122 [8/256/0/180] [14756.861937] LNet: Added LNI 192.168.202.110@tcp123 [8/256/0/180] [14757.765196] LNet: Added LNI 192.168.202.110@tcp124 [8/256/0/180] [14758.691939] LNet: Added LNI 192.168.202.110@tcp125 [8/256/0/180] [14759.622917] LNet: Added LNI 192.168.202.110@tcp126 [8/256/0/180] [14760.658658] LNet: Added LNI 192.168.202.110@tcp127 [8/256/0/180] [14761.616308] LNet: Added LNI 192.168.202.110@tcp128 [8/256/0/180] [14762.406993] LNet: Added LNI 192.168.202.110@tcp129 [8/256/0/180] [14763.171044] LNet: Added LNI 192.168.202.110@tcp130 [8/256/0/180] [14763.988869] LNet: Added LNI 192.168.202.110@tcp131 [8/256/0/180] [14764.932508] LNet: Added LNI 192.168.202.110@tcp132 [8/256/0/180] [14765.894987] LNet: Added LNI 192.168.202.110@tcp133 [8/256/0/180] [14766.804150] LNet: Added LNI 192.168.202.110@tcp134 [8/256/0/180] [14767.905565] LNet: Added LNI 192.168.202.110@tcp135 [8/256/0/180] [14775.199896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [14775.239258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14776.378941] Lustre: 464791:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14776.477681] LNetError: 462237:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14776.535953] LNetError: Protocol error connecting to 192.168.202.110@tcp120 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14776.559354] LNetError: Skipped 3 previous similar messages [14776.612174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14776.668431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14776.772764] Lustre: lustre-MDT0000: new disk, initializing [14776.884814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14776.911601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14777.184785] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp120 [14777.188258] LNetError: 464818:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14777.206560] LNetError: 464818:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14777.216570] LNetError: Protocol error connecting to 192.168.202.110@tcp104 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14779.242862] LNetError: 464818:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14779.257236] LNetError: 464818:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14779.266899] LNetError: Protocol error connecting to 192.168.202.110@tcp120 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14779.273356] LNetError: Skipped 5 previous similar messages [14781.733979] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14783.338027] LNetError: 464812:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14783.374737] LNetError: Protocol error connecting to 192.168.202.110@tcp120 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14791.523522] LNetError: 462239:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14791.528507] LNetError: Protocol error connecting to 192.168.202.110@tcp120 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14793.873597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14794.083904] Lustre: 464807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14794.144661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14794.150429] Lustre: Skipped 1 previous similar message [14794.275959] Lustre: lustre-MDT0001: new disk, initializing [14794.341139] Lustre: 465611:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14794.349856] Lustre: 465611:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14794.465393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14794.533219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14794.543567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14801.030992] Lustre: 465847:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14801.043632] Lustre: 465847:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14801.234450] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14807.975767] LNetError: 464817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14807.991852] LNetError: Protocol error connecting to 192.168.202.110@tcp120 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14813.319951] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14821.810541] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14824.356437] LNetError: 464817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14824.374749] LNetError: 464817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14824.385058] LNetError: Protocol error connecting to 192.168.202.110@tcp108 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14824.396743] LNetError: Skipped 8 previous similar messages [14830.851179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14831.287497] Lustre: lustre-OST0000: new disk, initializing [14831.290836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14831.299550] Lustre: 466784:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14831.411307] Lustre: 466789:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14831.420152] Lustre: 466789:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14831.485320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14837.669159] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp120 [14837.807473] Lustre: 467070:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14837.813872] Lustre: 467070:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14837.905292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14837.912473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14838.094751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14839.320482] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14850.338268] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14860.192323] LNetError: 464811:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14860.213184] LNetError: 464811:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 15 previous similar messages [14860.222389] LNetError: Protocol error connecting to 192.168.202.110@tcp118 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14860.237800] LNetError: Skipped 15 previous similar messages [14863.840204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14863.843110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14863.850304] LustreError: Skipped 1 previous similar message [14863.861547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14863.872306] Lustre: Skipped 1 previous similar message [14865.349221] Lustre: server umount lustre-OST0000 complete [14871.520580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14871.531264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14871.554148] Lustre: Skipped 1 previous similar message [14871.559285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14874.081651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14874.082105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14879.200556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14879.206474] Lustre: Skipped 2 previous similar messages [14884.319915] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14884.325350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14884.351077] Lustre: Skipped 1 previous similar message [14884.553979] Lustre: server umount lustre-MDT0000 complete [14889.104149] LustreError: 464800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986293 with bad export cookie 3065469287557211903 [14889.104847] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14889.129844] LustreError: 464800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14889.543458] Lustre: server umount lustre-MDT0001 complete [14902.149346] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14905.947878] Key type lgssc unregistered [14906.503930] LNet: 468699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14906.526486] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14906.530603] LNet: Removed LNI 192.168.202.110@tcp [14907.551868] LNet: Removed LNI 192.168.202.110@tcp100 [14907.557424] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14907.570334] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14907.581418] LNet: Removed LNI 192.168.202.110@tcp101 [14908.639473] LNet: Removed LNI 192.168.202.110@tcp102 [14908.673428] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14908.686362] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14908.698584] LNet: Removed LNI 192.168.202.110@tcp103 [14909.727656] LNet: Removed LNI 192.168.202.110@tcp104 [14910.815339] LNet: Removed LNI 192.168.202.110@tcp105 [14910.821352] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14910.833595] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14910.839414] LNet: Removed LNI 192.168.202.110@tcp106 [14911.904545] LNet: Removed LNI 192.168.202.110@tcp107 [14912.927139] LNet: Removed LNI 192.168.202.110@tcp108 [14913.951251] LNet: Removed LNI 192.168.202.110@tcp109 [14914.976429] LNet: Removed LNI 192.168.202.110@tcp110 [14914.984638] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14915.002946] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14915.025420] LNet: Removed LNI 192.168.202.110@tcp111 [14916.063676] LNet: Removed LNI 192.168.202.110@tcp112 [14917.087225] LNet: Removed LNI 192.168.202.110@tcp113 [14918.114522] LNet: Removed LNI 192.168.202.110@tcp114 [14919.135251] LNet: Removed LNI 192.168.202.110@tcp115 [14920.165390] LNet: Removed LNI 192.168.202.110@tcp116 [14921.183189] LNet: Removed LNI 192.168.202.110@tcp117 [14922.209375] LNet: Removed LNI 192.168.202.110@tcp118 [14923.234967] LNet: Removed LNI 192.168.202.110@tcp119 [14923.238820] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14923.245821] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14923.256964] LNet: Removed LNI 192.168.202.110@tcp120 [14924.319165] LNet: Removed LNI 192.168.202.110@tcp121 [14925.343174] LNet: Removed LNI 192.168.202.110@tcp122 [14926.368099] LNet: Removed LNI 192.168.202.110@tcp123 [14927.392949] LNet: Removed LNI 192.168.202.110@tcp124 [14928.417091] LNet: Removed LNI 192.168.202.110@tcp125 [14929.439324] LNet: Removed LNI 192.168.202.110@tcp126 [14930.463155] LNet: Removed LNI 192.168.202.110@tcp127 [14931.487172] LNet: Removed LNI 192.168.202.110@tcp128 [14932.511327] LNet: Removed LNI 192.168.202.110@tcp129 [14932.519231] LNet: Removed LNI 192.168.202.110@tcp130 [14932.528772] LNet: Removed LNI 192.168.202.110@tcp131 [14932.533915] LNet: Removed LNI 192.168.202.110@tcp132 [14932.541721] LNet: Removed LNI 192.168.202.110@tcp133 [14932.547519] LNet: Removed LNI 192.168.202.110@tcp134 [14932.560356] LNet: Removed LNI 192.168.202.110@tcp135 [14933.526645] Key type .llcrypt unregistered [14933.528850] Key type ._llcrypt unregistered [14962.928579] Key type ._llcrypt registered [14962.933401] Key type .llcrypt registered [14963.173074] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [14979.171699] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14980.578879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14980.827504] alg: No test for adler32 (adler32-zlib) [14981.854384] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [14982.225580] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [14984.056493] Key type lgssc registered [14985.446426] Lustre: Echo OBD driver; http://www.lustre.org/ [15025.924977] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:13:49 (1790986429) [15043.185846] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [15045.481785] Key type lgssc unregistered [15045.718818] LNet: 472804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15045.724082] LNetError: 472804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15045.742619] LNet: Removed LNI 192.168.202.110@tcp [15046.980367] Key type .llcrypt unregistered [15046.982743] Key type ._llcrypt unregistered [15065.933888] Key type ._llcrypt registered [15065.935990] Key type .llcrypt registered [15066.035871] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15067.525298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15067.625649] alg: No test for adler32 (adler32-zlib) [15069.012896] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [15069.446902] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [15071.311993] Key type lgssc registered [15072.941235] Lustre: Echo OBD driver; http://www.lustre.org/ [15084.092537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15084.131663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15085.458148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15085.497279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15085.681066] Lustre: lustre-MDT0000: new disk, initializing [15085.916286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15085.966158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15089.976851] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15101.103349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15101.297235] Lustre: 474030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15101.349578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15101.353170] Lustre: Skipped 1 previous similar message [15101.413425] Lustre: lustre-MDT0001: new disk, initializing [15101.554118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15101.605214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15101.627282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15107.430922] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15117.889577] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15124.299725] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15132.937626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15133.223862] Lustre: lustre-OST0000: new disk, initializing [15133.227572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15133.234765] Lustre: 476002:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15133.298945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15139.467333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15139.482139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15139.710628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15143.212648] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15156.090769] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15161.712905] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing lnet_if_list [15169.346777] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15169.395452] LNet: Added LNI 192.168.202.110@tcp10 [8/256/0/180] [15177.289376] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15177.308346] LNet: Added LNI 192.168.202.110@tcp11 [8/256/0/180] [15183.329609] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15183.378495] LNet: Added LNI 192.168.202.110@tcp12 [8/256/0/180] [15189.518818] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15189.553666] LNet: Added LNI 192.168.202.110@tcp13 [8/256/0/180] [15195.871206] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15195.888178] LNet: Added LNI 192.168.202.110@tcp14 [8/256/0/180] [15202.966971] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15202.995201] LNet: Added LNI 192.168.202.110@tcp15 [8/256/0/180] [15208.169223] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15208.222024] LNet: Added LNI 192.168.202.110@tcp16 [8/256/0/180] [15215.573879] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15215.596264] LNet: Added LNI 192.168.202.110@tcp17 [8/256/0/180] [15222.353458] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15222.406697] LNet: Added LNI 192.168.202.110@tcp18 [8/256/0/180] [15228.290209] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15228.310778] LNet: Added LNI 192.168.202.110@tcp19 [8/256/0/180] [15236.215711] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15236.268344] LNet: Added LNI 192.168.202.110@tcp20 [8/256/0/180] [15242.665345] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15242.686920] LNet: Added LNI 192.168.202.110@tcp21 [8/256/0/180] [15248.369379] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15248.407982] LNet: Added LNI 192.168.202.110@tcp22 [8/256/0/180] [15255.368390] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15255.418554] LNet: Added LNI 192.168.202.110@tcp23 [8/256/0/180] [15262.554902] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15262.581592] LNet: Added LNI 192.168.202.110@tcp24 [8/256/0/180] [15270.341405] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15270.398439] LNet: Added LNI 192.168.202.110@tcp25 [8/256/0/180] [15277.886718] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15277.929199] LNet: Added LNI 192.168.202.110@tcp26 [8/256/0/180] [15285.551964] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15285.628412] LNet: Added LNI 192.168.202.110@tcp27 [8/256/0/180] [15292.761123] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15292.792415] LNet: Added LNI 192.168.202.110@tcp28 [8/256/0/180] [15300.780824] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15300.846957] LNet: Added LNI 192.168.202.110@tcp29 [8/256/0/180] [15308.572077] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15308.661200] LNet: Added LNI 192.168.202.110@tcp30 [8/256/0/180] [15316.765263] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15316.800615] LNet: Added LNI 192.168.202.110@tcp31 [8/256/0/180] [15324.313835] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15324.358587] LNet: Added LNI 192.168.202.110@tcp32 [8/256/0/180] [15330.776784] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15330.798973] LNet: Added LNI 192.168.202.110@tcp33 [8/256/0/180] [15335.992639] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15336.039570] LNet: Added LNI 192.168.202.110@tcp34 [8/256/0/180] [15342.225208] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15342.267534] LNet: Added LNI 192.168.202.110@tcp35 [8/256/0/180] [15350.024522] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15350.052342] LNet: Added LNI 192.168.202.110@tcp36 [8/256/0/180] [15358.563419] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15358.608601] LNet: Added LNI 192.168.202.110@tcp37 [8/256/0/180] [15367.064150] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15367.098482] LNet: Added LNI 192.168.202.110@tcp38 [8/256/0/180] [15376.046256] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15376.080603] LNet: Added LNI 192.168.202.110@tcp39 [8/256/0/180] [15384.670518] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15384.710264] LNet: Added LNI 192.168.202.110@tcp40 [8/256/0/180] [15393.267901] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15393.311800] LNet: Added LNI 192.168.202.110@tcp41 [8/256/0/180] [15400.861358] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15400.899691] LNet: Added LNI 192.168.202.110@tcp42 [8/256/0/180] [15410.845245] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15410.907886] LNet: Added LNI 192.168.202.110@tcp43 [8/256/0/180] [15421.228926] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15421.286474] LNet: Added LNI 192.168.202.110@tcp44 [8/256/0/180] [15431.479352] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15431.531815] LNet: Added LNI 192.168.202.110@tcp45 [8/256/0/180] [15441.192544] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15441.230363] LNet: Added LNI 192.168.202.110@tcp46 [8/256/0/180] [15450.137896] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15450.164952] LNet: Added LNI 192.168.202.110@tcp47 [8/256/0/180] [15459.213869] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15459.238625] LNet: Added LNI 192.168.202.110@tcp48 [8/256/0/180] [15467.269703] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15467.316633] LNet: Added LNI 192.168.202.110@tcp49 [8/256/0/180] [15476.786341] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15476.813965] LNet: Added LNI 192.168.202.110@tcp50 [8/256/0/180] [15495.650161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15495.657170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15495.681880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15498.209694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15498.227680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15498.238817] Lustre: Skipped 1 previous similar message [15501.335620] Lustre: server umount lustre-MDT0000 complete [15504.573113] LustreError: 474022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790986908 with bad export cookie 1642501890750841388 [15504.616084] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15505.118939] Lustre: server umount lustre-MDT0001 complete [15524.832121] Lustre: 473294:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790986913/real 1790986913] req@000000005d6d94de x1877985432100736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790986929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15524.836359] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15524.880860] Lustre: 473294:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15528.852918] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15528.928538] Lustre: 473291:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790986917/real 1790986917] req@0000000007704621 x1877985432100864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790986933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15529.957258] Lustre: 473292:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790986918/real 1790986918] req@00000000b7207d50 x1877985432101120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790986934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15535.071115] Lustre: 473294:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790986923/real 1790986923] req@00000000d6138ad8 x1877985432101248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790986939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15539.218468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15549.746448] LustreError: 475995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15549.777786] LustreError: 475995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15549.831196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15550.815663] LustreError: 500251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15552.944698] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15552.955044] LustreError: 500249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15552.973719] LustreError: 500249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15554.255867] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15558.116553] LustreError: 500249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15558.134953] LustreError: 500249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15563.240571] LustreError: 500250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15563.275293] LustreError: 500250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15563.493418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15563.972634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15568.771423] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15569.397252] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15577.657601] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15583.629589] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15605.220406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15605.241819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15605.271061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15605.285420] Lustre: Skipped 2 previous similar messages [15609.761174] Lustre: server umount lustre-OST0000 complete [15614.944532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15614.951103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15614.960773] Lustre: Skipped 1 previous similar message [15614.968062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15614.979214] Lustre: Skipped 1 previous similar message [15619.501654] Lustre: server umount lustre-MDT0000 complete [15620.576554] LustreError: 500254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15620.600231] LustreError: 500254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15623.555995] LustreError: 474021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987027 with bad export cookie 1642501890750844055 [15623.562168] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15623.563367] LustreError: 474021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15623.932839] Lustre: server umount lustre-MDT0001 complete [15635.470405] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [15638.254983] Key type lgssc unregistered [15638.686339] LNet: 503049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15638.696276] LNetError: 503049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15638.704393] LNet: Removed LNI 192.168.202.110@tcp [15638.707306] LNet: Removed LNI 192.168.202.110@tcp10 [15638.711805] LNet: Removed LNI 192.168.202.110@tcp11 [15638.714577] LNet: Removed LNI 192.168.202.110@tcp12 [15638.718473] LNet: Removed LNI 192.168.202.110@tcp13 [15638.724091] LNet: Removed LNI 192.168.202.110@tcp14 [15638.728928] LNet: Removed LNI 192.168.202.110@tcp15 [15638.732601] LNet: Removed LNI 192.168.202.110@tcp16 [15638.739539] LNet: Removed LNI 192.168.202.110@tcp17 [15638.740974] LNet: Removed LNI 192.168.202.110@tcp18 [15638.742091] LNet: Removed LNI 192.168.202.110@tcp19 [15638.746107] LNet: Removed LNI 192.168.202.110@tcp20 [15638.757899] LNet: Removed LNI 192.168.202.110@tcp21 [15638.760301] LNet: Removed LNI 192.168.202.110@tcp22 [15638.763202] LNet: Removed LNI 192.168.202.110@tcp23 [15638.767831] LNet: Removed LNI 192.168.202.110@tcp24 [15638.777126] LNet: Removed LNI 192.168.202.110@tcp25 [15638.781667] LNet: Removed LNI 192.168.202.110@tcp26 [15638.791035] LNet: Removed LNI 192.168.202.110@tcp27 [15638.793127] LNet: Removed LNI 192.168.202.110@tcp28 [15638.805464] LNet: Removed LNI 192.168.202.110@tcp29 [15638.814678] LNet: Removed LNI 192.168.202.110@tcp30 [15638.816316] LNet: Removed LNI 192.168.202.110@tcp31 [15638.822319] LNet: Removed LNI 192.168.202.110@tcp32 [15638.824066] LNet: Removed LNI 192.168.202.110@tcp33 [15638.826636] LNet: Removed LNI 192.168.202.110@tcp34 [15638.828937] LNet: Removed LNI 192.168.202.110@tcp35 [15638.831580] LNet: Removed LNI 192.168.202.110@tcp36 [15638.834147] LNet: Removed LNI 192.168.202.110@tcp37 [15638.836279] LNet: Removed LNI 192.168.202.110@tcp38 [15638.838603] LNet: Removed LNI 192.168.202.110@tcp39 [15638.841754] LNet: Removed LNI 192.168.202.110@tcp40 [15638.843204] LNet: Removed LNI 192.168.202.110@tcp41 [15638.845630] LNet: Removed LNI 192.168.202.110@tcp42 [15638.849294] LNet: Removed LNI 192.168.202.110@tcp43 [15638.852556] LNet: Removed LNI 192.168.202.110@tcp44 [15638.855863] LNet: Removed LNI 192.168.202.110@tcp45 [15638.858300] LNet: Removed LNI 192.168.202.110@tcp46 [15638.861380] LNet: Removed LNI 192.168.202.110@tcp47 [15638.863573] LNet: Removed LNI 192.168.202.110@tcp48 [15638.869179] LNet: Removed LNI 192.168.202.110@tcp49 [15638.883562] LNet: Removed LNI 192.168.202.110@tcp50 [15639.683650] Key type .llcrypt unregistered [15639.685154] Key type ._llcrypt unregistered [15661.045827] Key type ._llcrypt registered [15661.047806] Key type .llcrypt registered [15661.146367] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [15675.716608] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15677.352861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15677.394045] alg: No test for adler32 (adler32-zlib) [15678.599314] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [15678.959429] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [15680.615421] Key type lgssc registered [15681.833559] Lustre: Echo OBD driver; http://www.lustre.org/ [15717.152438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15717.181812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15718.534175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15718.557170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15718.611580] Lustre: lustre-MDT0000: new disk, initializing [15718.665652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15718.680370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15723.233834] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15734.657501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15734.816217] Lustre: 506265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15734.904973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15734.908475] Lustre: Skipped 1 previous similar message [15734.987305] Lustre: lustre-MDT0001: new disk, initializing [15735.116790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15735.173062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15735.198372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15739.431477] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15748.362399] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15753.952878] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15760.787713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15760.972578] Lustre: *** cfs_fail_loc=90f, val=0*** [15761.013524] Lustre: lustre-OST0000: new disk, initializing [15761.016567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15761.021436] Lustre: 508279:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15761.091641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15766.033291] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15769.640470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15769.648046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15769.721045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15774.437414] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15784.933273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15784.934332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15784.945974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15784.955265] LustreError: Skipped 1 previous similar message [15784.968558] Lustre: Skipped 1 previous similar message [15787.865465] Lustre: server umount lustre-OST0000 complete [15791.584507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15791.587806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15791.603960] Lustre: Skipped 1 previous similar message [15791.608902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15795.173235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15795.174263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15795.206420] Lustre: Skipped 1 previous similar message [15797.815892] Lustre: server umount lustre-MDT0000 complete [15801.164563] LustreError: 506255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987205 with bad export cookie 12130747049347422517 [15801.171861] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15801.178469] LustreError: 506255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15801.537248] Lustre: server umount lustre-MDT0001 complete [15811.251628] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [15814.034739] Key type lgssc unregistered [15814.325204] LNet: 510185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15814.348106] LNetError: 510185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15814.382234] LNet: Removed LNI 192.168.202.110@tcp [15815.532679] Key type .llcrypt unregistered [15815.539129] Key type ._llcrypt unregistered [15834.679595] Key type ._llcrypt registered [15834.681682] Key type .llcrypt registered [15834.798622] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15836.038536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15836.393473] alg: No test for adler32 (adler32-zlib) [15837.635933] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [15838.018429] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [15839.767286] Key type lgssc registered [15840.841079] Lustre: Echo OBD driver; http://www.lustre.org/ [15848.953212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15848.974265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15850.618146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15855.145418] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15860.206664] LustreError: 511473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15860.231274] LustreError: 511473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15863.758750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15864.175873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15868.645355] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15874.801865] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15879.980567] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15886.226892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15886.594927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15893.785242] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15901.418530] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15912.046182] loop0: detected capacity change from 0 to 204800000 [15912.230727] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15912.242367] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15912.257131] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15917.946275] loop0: detected capacity change from 0 to 204800000 [15918.116478] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15918.126942] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15918.137186] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15922.814665] loop0: detected capacity change from 0 to 204800000 [15927.982981] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15928.156439] Lustre: fs15246-OST0000: new disk, initializing [15928.161791] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15928.169150] Lustre: 515065:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15928.231389] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15934.294575] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15943.412173] loop1: detected capacity change from 0 to 204800000 [15949.061532] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15949.179479] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15949.219882] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15949.305834] Lustre: fs15246-MDT0000: new disk, initializing [15949.516721] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15949.555684] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15949.732677] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15950.822194] Lustre: 516188:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f9111c80 x1877986238313216/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790987370 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15954.596355] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15959.010851] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15959.028059] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15959.078821] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15977.447608] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15977.457756] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15981.099406] Lustre: server umount fs15246-MDT0000 complete [15982.561027] LustreError: 514974:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15987.690658] LustreError: 511467:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15990.034643] Lustre: server umount fs15246-OST0000 complete [16002.539426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16002.549111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16002.568681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16007.437795] Lustre: server umount lustre-OST0000 complete [16012.772495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16012.780795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16012.794738] Lustre: Skipped 1 previous similar message [16012.811040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16012.817137] Lustre: Skipped 2 previous similar messages [16017.269963] Lustre: server umount lustre-MDT0000 complete [16018.400604] LustreError: 511473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16018.427580] LustreError: 511473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16020.798079] LustreError: 516182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987424 with bad export cookie 4313800714558616814 [16020.801755] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16020.812082] LustreError: 516182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16021.026518] Lustre: server umount lustre-MDT0001 complete [16031.223144] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [16033.968223] Key type lgssc unregistered [16034.269112] LNet: 519282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16034.277732] LNetError: 519282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16035.302782] LNet: Removed LNI 192.168.202.110@tcp [16036.040605] Key type .llcrypt unregistered [16036.042269] Key type ._llcrypt unregistered [16055.439693] Key type ._llcrypt registered [16055.441338] Key type .llcrypt registered [16055.561495] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16056.628968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16056.706356] alg: No test for adler32 (adler32-zlib) [16057.720626] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [16057.913770] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [16059.583194] Key type lgssc registered [16060.481649] Lustre: Echo OBD driver; http://www.lustre.org/ [16093.069759] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [16102.876523] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16135.173222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16135.198813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16136.446798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16136.472507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16136.544268] Lustre: lustre-MDT0000: new disk, initializing [16136.670587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16136.688139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16140.221503] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16150.167733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16150.263363] Lustre: 523764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16150.323885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16150.334489] Lustre: Skipped 1 previous similar message [16150.393539] Lustre: lustre-MDT0001: new disk, initializing [16150.496452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16150.528892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16150.535619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16154.907876] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16163.369582] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16168.257915] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16173.846317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16174.042599] Lustre: lustre-OST0000: new disk, initializing [16174.046051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16174.053318] Lustre: 525732:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16174.154181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16179.641516] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16179.756086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16179.767051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16179.945771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16188.627545] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16192.860852] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16193.025809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16196.885446] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16197.086386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16200.678283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16200.680037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16200.702652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16200.722924] Lustre: Skipped 1 previous similar message [16205.027135] Lustre: server umount lustre-OST0000 complete [16210.917627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16210.928370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16210.931844] Lustre: Skipped 1 previous similar message [16210.941914] Lustre: Skipped 2 previous similar messages [16214.756307] Lustre: server umount lustre-MDT0000 complete [16218.092864] LustreError: 523755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790987622 with bad export cookie 10316795239938586215 [16218.096728] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16218.108219] LustreError: 523755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16218.374368] Lustre: server umount lustre-MDT0001 complete [16242.780112] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [16252.985933] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16280.850180] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16035 sec ========= 20:34:44 (1790987684) [16282.373179] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:34:46 (1790987686) === [16285.412695] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:34:49 (1790987689) === [16316.732255] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [16318.865495] Key type lgssc unregistered [16319.147888] LNet: 532370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16319.154763] LNetError: 532370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16319.178563] LNet: Removed LNI 192.168.202.110@tcp [16319.814315] Key type .llcrypt unregistered [16319.817907] Key type ._llcrypt unregistered