[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 543368381 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002313] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007480] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010126] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.020553] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024816] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026091] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028098] *** VALIDATE tmpfs *** [ 0.029030] *** VALIDATE proc *** [ 0.031258] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033240] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037024] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.040563] debug: unmapping init [mem 0xffffffffb6659000-0xffffffffb6660fff] [ 0.043143] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044622] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048009] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052295] rcu: Hierarchical SRCU implementation. [ 0.054480] smp: Bringing up secondary CPUs ... [ 0.055540] x86: Booting SMP configuration: [ 0.056024] .... node #0, CPUs: #1 #2 #3 [ 0.059441] smp: Brought up 1 node, 4 CPUs [ 0.061010] smpboot: Max logical packages: 1 [ 0.062016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132169] node 0 deferred pages initialised in 65ms [ 0.134105] devtmpfs: initialized [ 0.136263] x86/mm: Memory block size: 128MB [ 0.140051] gcov: version magic: 0x41383552 [ 0.143370] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148311] pinctrl core: initialized pinctrl subsystem [ 0.150181] [ 0.150744] ************************************************************* [ 0.154010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157011] ** ** [ 0.159010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161010] ** ** [ 0.163008] ** This means that this kernel is built to expose internal ** [ 0.165010] ** IOMMU data structures, which may compromise security on ** [ 0.167009] ** your system. ** [ 0.169010] ** ** [ 0.172010] ** If you see this message and you are not debugging the ** [ 0.174009] ** kernel, report this immediately to your vendor! ** [ 0.175007] ** ** [ 0.177009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179016] ************************************************************* [ 0.181612] NET: Registered protocol family 16 [ 0.183489] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.186061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.191082] cpuidle: using governor menu [ 0.192818] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194536] PCI: Using configuration type 1 for base access [ 0.195129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208104] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212061] cryptd: max_cpu_qlen set to 1000 [ 0.215258] ACPI: Added _OSI(Module Device) [ 0.217015] ACPI: Added _OSI(Processor Device) [ 0.219011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226299] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233072] ACPI: Interpreter enabled [ 0.234057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.236009] ACPI: Using IOAPIC for interrupt routing [ 0.237088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.240388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.248000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.251026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.253012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.255151] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259430] acpiphp: Slot [2] registered [ 0.261108] acpiphp: Slot [5] registered [ 0.263125] acpiphp: Slot [6] registered [ 0.264150] acpiphp: Slot [7] registered [ 0.266102] acpiphp: Slot [8] registered [ 0.268109] acpiphp: Slot [9] registered [ 0.270119] acpiphp: Slot [10] registered [ 0.271103] acpiphp: Slot [3] registered [ 0.273157] acpiphp: Slot [4] registered [ 0.275086] acpiphp: Slot [11] registered [ 0.276134] acpiphp: Slot [12] registered [ 0.278101] acpiphp: Slot [13] registered [ 0.280111] acpiphp: Slot [14] registered [ 0.282082] acpiphp: Slot [15] registered [ 0.283163] acpiphp: Slot [16] registered [ 0.285078] acpiphp: Slot [17] registered [ 0.288100] acpiphp: Slot [18] registered [ 0.289086] acpiphp: Slot [19] registered [ 0.293097] acpiphp: Slot [20] registered [ 0.294142] acpiphp: Slot [21] registered [ 0.295182] acpiphp: Slot [22] registered [ 0.297082] acpiphp: Slot [23] registered [ 0.298062] acpiphp: Slot [24] registered [ 0.299065] acpiphp: Slot [25] registered [ 0.300068] acpiphp: Slot [26] registered [ 0.301084] acpiphp: Slot [27] registered [ 0.302068] acpiphp: Slot [28] registered [ 0.304082] acpiphp: Slot [29] registered [ 0.305086] acpiphp: Slot [30] registered [ 0.306059] acpiphp: Slot [31] registered [ 0.307042] PCI host bridge to bus 0000:00 [ 0.308013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.310015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321149] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.323882] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.326127] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.335012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.339047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.343016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.347592] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.350861] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.353041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356886] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.375929] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.381014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.389283] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.399016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.406017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.443015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452936] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.545017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.556123] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.566025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.578018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.606009] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.625727] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.639019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.649020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.723016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.747822] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.783066] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.807017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.854023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.891368] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.899014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.912019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.963017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.987253] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.988283] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.990341] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.992255] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.993142] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.998091] iommu: Default domain type: Passthrough [ 0.999424] SCSI subsystem initialized [ 1.000107] ACPI: bus type USB registered [ 1.001083] usbcore: registered new interface driver usbfs [ 1.002161] usbcore: registered new interface driver hub [ 1.004088] usbcore: registered new device driver usb [ 1.006139] pps_core: LinuxPPS API ver. 1 registered [ 1.008010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.011056] PTP clock support registered [ 1.013062] EDAC MC: Ver: 3.0.0 [ 1.015426] PCI: Using ACPI for IRQ routing [ 1.016743] NetLabel: Initializing [ 1.018009] NetLabel: domain hash size = 128 [ 1.019006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.020131] NetLabel: unlabeled traffic allowed by default [ 1.022121] vgaarb: loaded [ 1.024218] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.026012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.034000] clocksource: Switched to clocksource kvm-clock [ 1.148656] VFS: Disk quotas dquot_6.6.0 [ 1.150407] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.152501] *** VALIDATE ramfs *** [ 1.153555] *** VALIDATE hugetlbfs *** [ 1.155817] pnp: PnP ACPI init [ 1.157932] pnp: PnP ACPI: found 6 devices [ 1.173815] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.176573] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.178292] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.179919] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.181845] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.183701] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.186054] NET: Registered protocol family 2 [ 1.188235] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.192761] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.195647] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.199917] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.202679] TCP: Hash tables configured (established 65536 bind 65536) [ 1.204863] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.207268] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.209402] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.211520] NET: Registered protocol family 1 [ 1.214292] RPC: Registered named UNIX socket transport module. [ 1.216208] RPC: Registered udp transport module. [ 1.217547] RPC: Registered tcp transport module. [ 1.219517] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.221346] NET: Registered protocol family 44 [ 1.222704] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.224480] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.226169] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.228234] PCI: CLS 0 bytes, default 64 [ 1.229845] Unpacking initramfs... [ 2.610158] debug: unmapping init [mem 0xffff9b24bcc54000-0xffff9b24bffbffff] [ 2.613987] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.616428] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.619440] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.100070] Initialise system trusted keyrings [ 3.101326] Key type blacklist registered [ 3.102901] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.111116] zbud: loaded [ 3.114114] *** VALIDATE nfs *** [ 3.115090] *** VALIDATE nfs4 *** [ 3.116256] pstore: using deflate compression [ 3.119353] Platform Keyring initialized [ 3.237523] NET: Registered protocol family 38 [ 3.238895] Key type asymmetric registered [ 3.240361] Asymmetric key parser 'x509' registered [ 3.241814] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.244198] io scheduler mq-deadline registered [ 3.246030] io scheduler kyber registered [ 3.247525] io scheduler bfq registered [ 3.250041] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.252844] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.255603] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.258422] ACPI: Power Button [PWRF] [ 3.263542] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.269565] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.290545] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.301565] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.315523] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.353728] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.382748] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.387383] Non-volatile memory driver v1.3 [ 3.388471] Linux agpgart interface v0.103 [ 3.420335] virtio_blk virtio1: [vda] 146728 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.423441] vda: detected capacity change from 0 to 75124736 [ 3.442305] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.445244] vdb: detected capacity change from 0 to 1073741824 [ 3.461220] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.464348] vdc: detected capacity change from 0 to 2621440000 [ 3.478149] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.480658] vdd: detected capacity change from 0 to 2621440000 [ 3.497172] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.499456] vde: detected capacity change from 0 to 4294967296 [ 3.513481] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.515367] vdf: detected capacity change from 0 to 4294967296 [ 3.521884] libphy: Fixed MDIO Bus: probed [ 3.526300] usbcore: registered new interface driver usbserial_generic [ 3.527693] usbserial: USB Serial support registered for generic [ 3.530089] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.534256] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.535601] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.537646] mousedev: PS/2 mouse device common for all mice [ 3.539664] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.542316] rtc_cmos 00:05: RTC can wake from S4 [ 3.547240] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.547308] rtc_cmos 00:05: registered as rtc0 [ 3.551347] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.551708] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.554907] intel_pstate: CPU model not supported [ 3.559548] hid: raw HID events driver (C) Jiri Kosina [ 3.561422] usbcore: registered new interface driver usbhid [ 3.562780] usbhid: USB HID core driver [ 3.563825] drop_monitor: Initializing network drop monitor service [ 3.565298] Initializing XFRM netlink socket [ 3.566544] NET: Registered protocol family 10 [ 3.568702] Segment Routing with IPv6 [ 3.569792] NET: Registered protocol family 17 [ 3.571437] mpls_gso: MPLS GSO support [ 3.575395] RAS: Correctable Errors collector initialized. [ 3.576765] AVX version of gcm_enc/dec engaged. [ 3.578635] AES CTR mode by8 optimization enabled [ 3.633904] sched_clock: Marking stable (3633890773, 0)->(4591094225, -957203452) [ 3.636834] registered taskstats version 1 [ 3.638205] Loading compiled-in X.509 certificates [ 3.639398] zswap: loaded using pool lzo/zbud [ 3.656421] Key type big_key registered [ 3.665533] Key type encrypted registered [ 3.666622] ima: No TPM chip found, activating TPM-bypass! [ 3.667970] ima: Allocated hash algorithm: sha1 [ 3.668858] ima: No architecture policies found [ 3.669858] evm: Initialising EVM extended attributes: [ 3.671177] evm: security.selinux [ 3.671859] evm: security.ima [ 3.672625] evm: security.capability [ 3.673455] evm: HMAC attrs: 0x1 [ 3.674917] rtc_cmos 00:05: setting system clock to 2026-09-08 04:28:11 UTC (1788841691) [ 3.679521] debug: unmapping init [mem 0xffffffffb7603000-0xffffffffb77fffff] [ 3.681530] debug: unmapping init [mem 0xffffffffb6382000-0xffffffffb6658fff] [ 3.690094] Write protecting the kernel read-only data: 28672k [ 3.692251] debug: unmapping init [mem 0xffffffffb4a03000-0xffffffffb4bfffff] [ 3.695082] debug: unmapping init [mem 0xffffffffb5314000-0xffffffffb53fffff] [ 3.723116] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.728551] systemd[1]: Detected virtualization kvm. [ 3.729800] systemd[1]: Detected architecture x86-64. [ 3.731109] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.753572] systemd[1]: No hostname configured. [ 3.754554] systemd[1]: Set hostname to . [ 3.755734] random: systemd: uninitialized urandom read (16 bytes read) [ 3.757255] systemd[1]: Initializing machine ID from random generator. [ 3.808532] random: ln: uninitialized urandom read (6 bytes read) [ 3.875493] random: systemd: uninitialized urandom read (16 bytes read) [ 3.878465] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.882814] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.887161] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.393784] device-mapper: uevent: version 1.0.3 [ 4.395408] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System...[ 4.940503] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.112448] virtio_net virtio0 ens2: renamed from eth0 [ 5.190197] scsi host0: ata_piix [ 5.213212] scsi host1: ata_piix [ 5.214936] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.217345] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.735290] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.176956] random: crng init done [ 10.178420] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.697546] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.889598] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.211244] SELinux: Disabled at runtime. [ 12.275654] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.286334] systemd[1]: Detected virtualization kvm. [ 12.288841] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.749405] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.751797] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.755726] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.759502] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.762036] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.774395] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.780185] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Reques[ 12.863515] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ts to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.279790] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.522355] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.536049] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.799035] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.814267] EDAC sbridge: Ver: 1.1.2 [ 15.916443] Key type dns_resolver registered [ 16.220543] NFS: Registering the id_resolver key type [ 16.222545] Key type id_resolver registered [ 16.224364] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg217-server login: [ 44.137173] libcfs: loading out-of-tree module taints kernel. [ 44.236723] Key type ._llcrypt registered [ 44.238670] Key type .llcrypt registered [ 44.390406] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 63.399435] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 65.115137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.129796] alg: No test for adler32 (adler32-zlib) [ 66.572251] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 67.445244] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 69.231195] Key type lgssc registered [ 70.784679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.926100] hrtimer: interrupt took 10041856 ns [ 87.539706] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.211313] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 140.248525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 140.295978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.865247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 141.954919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.103882] Lustre: lustre-MDT0000: new disk, initializing [ 142.258079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.281218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.537733] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.164976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.315178] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.359222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.363063] Lustre: Skipped 1 previous similar message [ 160.444580] Lustre: lustre-MDT0001: new disk, initializing [ 160.517726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.541243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 160.551403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.284668] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.664337] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 179.687142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.912309] Lustre: lustre-OST0000: new disk, initializing [ 179.915979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 179.922322] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.038191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.721992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 180.740978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 180.847397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 187.155191] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.242365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.412647] Lustre: lustre-OST0001: new disk, initializing [ 202.417856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 202.426575] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.543555] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.349451] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.484197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 209.505146] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 209.606490] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.917465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.640532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.083583] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.076837] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing yml_node [ 244.377323] Lustre: DEBUG MARKER: Client: 2.17.57.114 [ 246.953259] Lustre: DEBUG MARKER: MDS: 2.17.57.114 [ 249.409241] Lustre: DEBUG MARKER: OSS: 2.17.57.114 [ 251.253529] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 00:32:18 EDT 2026 [ 268.823382] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 270.425984] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 286.693068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.699639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.710522] Lustre: Skipped 1 previous similar message [ 286.718053] Lustre: Skipped 3 previous similar messages [ 291.819168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.830527] Lustre: Skipped 3 previous similar messages [ 292.190724] Lustre: server umount lustre-MDT0000 complete [ 295.312606] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841983 with bad export cookie 8866148795330866119 [ 295.318841] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.321350] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 296.931887] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.938154] Lustre: Skipped 3 previous similar messages [ 296.940672] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.960147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.969029] Lustre: Skipped 2 previous similar messages [ 301.727418] Lustre: server umount lustre-MDT0001 complete [ 305.569659] Lustre: server umount lustre-OST0000 complete [ 308.783667] Lustre: server umount lustre-OST0001 complete [ 315.129796] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 325.972554] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 360.334092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.614890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 360.643753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 360.712536] Lustre: lustre-MDT0000: new disk, initializing [ 360.842256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.858555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 365.654265] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 376.172429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.267785] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 376.346525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 376.349508] Lustre: Skipped 1 previous similar message [ 376.410235] Lustre: lustre-MDT0001: new disk, initializing [ 376.653993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 376.728574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 376.752180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 381.334556] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 391.400576] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 397.777859] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 404.744305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 405.125487] Lustre: lustre-OST0000: new disk, initializing [ 405.128145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 405.133094] Lustre: 17038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 405.328284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.770335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 406.785912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 406.902925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 412.467445] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 421.539885] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 426.123863] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 426.303267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 431.102933] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 431.303746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 437.728243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 437.731912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.735737] LustreError: Skipped 1 previous similar message [ 437.737072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 437.747380] Lustre: Skipped 1 previous similar message [ 439.521169] Lustre: server umount lustre-OST0000 complete [ 443.361093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 443.369585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.970700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.971518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 448.003924] Lustre: Skipped 3 previous similar messages [ 449.546896] Lustre: server umount lustre-MDT0000 complete [ 453.089792] LustreError: 15956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.111884] LustreError: 15956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 454.239078] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842142 with bad export cookie 8866148795330867652 [ 454.251615] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.254468] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 454.855482] Lustre: server umount lustre-MDT0001 complete [ 460.578515] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:35:46 (1788842146) [ 463.715817] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 466.033729] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:35:52 (1788842152) [ 472.805416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.186810] LustreError: 19450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.247965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.460227] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 478.690885] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.810821] LustreError: 19450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.006321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.336696] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 500.284472] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 506.843847] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 513.607876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 520.081404] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 528.146603] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 539.634388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.637972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.652368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.657928] Lustre: Skipped 1 previous similar message [ 539.676264] Lustre: Skipped 1 previous similar message [ 542.335903] Lustre: server umount lustre-OST0000 complete [ 552.537735] Lustre: server umount lustre-MDT0000 complete [ 554.985110] LustreError: 20178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.005877] LustreError: 20178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 556.522986] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842244 with bad export cookie 8866148795330868534 [ 556.545672] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.983270] Lustre: server umount lustre-MDT0001 complete [ 567.155571] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 570.339682] Key type lgssc unregistered [ 570.735435] LNet: 22980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.756333] LNetError: 22980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.793262] LNet: Removed LNI 192.168.202.117@tcp [ 572.163293] Key type .llcrypt unregistered [ 572.165579] Key type ._llcrypt unregistered [ 597.007847] Key type ._llcrypt registered [ 597.014834] Key type .llcrypt registered [ 597.185505] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 598.647827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 598.765713] alg: No test for adler32 (adler32-zlib) [ 599.883899] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 600.322363] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 602.079179] Key type lgssc registered [ 603.427917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 612.429908] loop: module loaded [ 612.564107] loop0: detected capacity change from 0 to 204800000 [ 612.764866] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 612.786297] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 613.127413] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.679693] loop0: detected capacity change from 0 to 204800000 [ 619.909951] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.921505] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.939683] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 625.457304] loop0: detected capacity change from 0 to 204800000 [ 630.632068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 630.649221] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.968822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 632.038909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 632.121577] Lustre: lustre-MDT0000: new disk, initializing [ 632.224102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 632.271796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 637.224494] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 646.051819] loop1: detected capacity change from 0 to 204800000 [ 651.763431] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 651.945255] Lustre: lustre-OST07c6: new disk, initializing [ 651.948897] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 651.954177] Lustre: Skipped 1 previous similar message [ 651.959759] Lustre: 26164:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 652.039070] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 657.982872] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 660.113157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 660.126754] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 660.187103] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 662.503637] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 662.521423] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 662.533875] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 662.539805] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 662.554539] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 662.566510] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 678.662970] Lustre: 25150:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 678.684681] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 678.695809] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 678.703955] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 678.711878] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 678.725970] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.672561] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 683.684490] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 683.706668] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 683.724819] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.744647] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 683.754921] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.761615] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 683.768634] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.786497] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 683.790833] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.795729] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.800310] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.170733] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 689.199555] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 689.215920] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 689.223234] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.227836] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 689.235741] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.243442] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 689.258659] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.270017] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 689.279888] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.291968] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 689.309325] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.897450] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 696.910948] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 696.914701] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 696.918465] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.921943] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.925617] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.929195] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 696.932091] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.935067] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 696.937932] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.941775] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.955609] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 707.590614] Lustre: 25150:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 707.604681] Lustre: 25150:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 707.614775] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 707.625644] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 707.634157] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.643723] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 707.652762] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 707.663070] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 707.677241] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.692887] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 707.704624] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.712356] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 752.098180] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.121102] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 757.246821] Lustre: server umount lustre-OST07c6 complete [ 766.395944] Lustre: server umount lustre-MDT0000 complete [ 780.557551] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 782.972836] Key type lgssc unregistered [ 783.218597] LNet: 28742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.225548] LNetError: 28742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.242205] LNet: Removed LNI 192.168.202.117@tcp [ 784.029171] Key type .llcrypt unregistered [ 784.032087] Key type ._llcrypt unregistered [ 802.841719] Key type ._llcrypt registered [ 802.843430] Key type .llcrypt registered [ 802.934717] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 804.414477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.466073] alg: No test for adler32 (adler32-zlib) [ 805.493754] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 805.740262] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 807.439202] Key type lgssc registered [ 808.764838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 817.031442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 817.051135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.395393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 822.829292] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 828.901472] LustreError: 30009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.922622] LustreError: 30009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 832.044076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.545355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 836.860928] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 844.346131] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 850.181639] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 856.868594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.301457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.491166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 863.903729] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 870.317795] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 884.192845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.200822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.202848] LustreError: Skipped 1 previous similar message [ 884.207936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.214304] Lustre: Skipped 1 previous similar message [ 884.230776] Lustre: Skipped 1 previous similar message [ 889.193949] Lustre: server umount lustre-OST0000 complete [ 893.921197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 893.931920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.946251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 899.552752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 899.562975] Lustre: Skipped 3 previous similar messages [ 904.673059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.684398] Lustre: Skipped 1 previous similar message [ 907.231146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 907.473818] Lustre: server umount lustre-MDT0000 complete [ 909.794677] LustreError: 31667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 909.810323] LustreError: 31667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 911.536339] LustreError: 29988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842599 with bad export cookie 1139079855399662659 [ 911.542367] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 911.959741] Lustre: server umount lustre-MDT0001 complete [ 924.401644] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 927.107881] Key type lgssc unregistered [ 927.451136] LNet: 33744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.468667] LNetError: 33744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.489297] LNet: Removed LNI 192.168.202.117@tcp [ 929.552738] Key type .llcrypt unregistered [ 929.555086] Key type ._llcrypt unregistered [ 949.281872] Key type ._llcrypt registered [ 949.284995] Key type .llcrypt registered [ 949.422344] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 950.458424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 950.517311] alg: No test for adler32 (adler32-zlib) [ 951.517986] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 951.727304] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 953.391171] Key type lgssc registered [ 954.478676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 962.252505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 962.274412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.788869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.164315] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 973.285490] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.306973] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 976.980668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.366088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.095835] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 989.858785] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 994.402293] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1002.092432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1002.469996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1008.704423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1009.589308] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1016.977682] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1029.093807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.094981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1029.100418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.103312] Lustre: Skipped 1 previous similar message [ 1029.126456] LustreError: Skipped 1 previous similar message [ 1033.603961] Lustre: server umount lustre-OST0000 complete [ 1038.822042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1038.830429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.847362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1038.856618] Lustre: Skipped 1 previous similar message [ 1043.836349] Lustre: server umount lustre-MDT0000 complete [ 1044.449490] LustreError: 36445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.472054] LustreError: 36445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1047.415472] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842735 with bad export cookie 17904749587527127637 [ 1047.423883] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1047.699328] Lustre: server umount lustre-MDT0001 complete [ 1058.122494] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1061.856504] Key type lgssc unregistered [ 1062.269787] LNet: 38543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.299440] LNetError: 38543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.313351] LNet: Removed LNI 192.168.202.117@tcp [ 1063.676361] Key type .llcrypt unregistered [ 1063.677696] Key type ._llcrypt unregistered [ 1089.979781] Key type ._llcrypt registered [ 1089.984032] Key type .llcrypt registered [ 1090.135993] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1091.401461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1091.744090] alg: No test for adler32 (adler32-zlib) [ 1093.125394] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 1093.703082] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1095.447217] Key type lgssc registered [ 1096.628899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1105.094337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1105.136661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.883854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.171319] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1117.152865] LustreError: 39959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.190301] LustreError: 39959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1121.296668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.646278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1126.136394] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1134.317289] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1139.709581] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1147.764655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1148.206203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.321917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1154.133899] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1161.078870] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1171.940430] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.944041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1171.956035] Lustre: Skipped 1 previous similar message [ 1171.977282] Lustre: Skipped 2 previous similar messages [ 1172.975805] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 1176.217583] Lustre: server umount lustre-MDT0000 complete [ 1177.072535] LustreError: 39952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.110690] LustreError: 39952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1179.853558] LustreError: 39941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842867 with bad export cookie 6550033147075627884 [ 1179.869101] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.875507] LustreError: 39941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1182.181712] LustreError: 41199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.186459] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.207905] LustreError: 41199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1182.242630] Lustre: Skipped 1 previous similar message [ 1182.255526] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1184.230348] LustreError: 39954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.236828] LustreError: 39954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1186.195813] Lustre: server umount lustre-MDT0001 complete [ 1207.248988] Lustre: server umount lustre-OST0000 complete [ 1224.789157] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1227.983613] Key type lgssc unregistered [ 1228.236843] LNet: 43818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.247299] LNetError: 43818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.277428] LNet: Removed LNI 192.168.202.117@tcp [ 1229.237615] Key type .llcrypt unregistered [ 1229.239647] Key type ._llcrypt unregistered [ 1255.392467] Key type ._llcrypt registered [ 1255.396733] Key type .llcrypt registered [ 1255.526573] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1256.671493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1256.739389] alg: No test for adler32 (adler32-zlib) [ 1257.833479] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 1258.049900] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1259.759258] Key type lgssc registered [ 1261.049792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1267.707605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1267.726505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.289676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1272.934486] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1279.465975] LustreError: 45231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.491960] LustreError: 45231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1280.979338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.243849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1285.490595] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1292.939662] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1298.995219] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1305.753536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1306.332510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.662634] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1314.562950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1321.405937] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1328.622650] Lustre: lustre-OST0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 1330.144978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1330.151557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1330.167154] Lustre: Skipped 1 previous similar message [ 1330.180413] Lustre: Skipped 1 previous similar message [ 1333.713383] Lustre: lustre-OST0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 1334.392355] Lustre: server umount lustre-OST0000 complete [ 1353.185918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1353.198918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.214507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.749038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.881762] Lustre: server umount lustre-MDT0000 complete [ 1360.886687] LustreError: 45230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.916203] LustreError: 45230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1362.326536] LustreError: 45966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843050 with bad export cookie 7690174195610008452 [ 1362.333864] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.340381] LustreError: 45966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1368.770607] Lustre: server umount lustre-MDT0001 complete [ 1380.065983] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1382.912579] Key type lgssc unregistered [ 1383.259948] LNet: 48939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.266050] LNetError: 48939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.279218] LNet: Removed LNI 192.168.202.117@tcp [ 1384.253524] Key type .llcrypt unregistered [ 1384.254660] Key type ._llcrypt unregistered [ 1406.869558] Key type ._llcrypt registered [ 1406.873629] Key type .llcrypt registered [ 1406.995606] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1408.246484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1408.502168] alg: No test for adler32 (adler32-zlib) [ 1409.567774] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 1409.882654] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1411.568519] Key type lgssc registered [ 1412.775453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1420.451585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1420.467876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.974935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.667022] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1431.521269] LustreError: 50364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.529299] LustreError: 50364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1435.650533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.297636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1440.480073] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1448.452151] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1453.910805] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1463.340742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1463.778895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.904668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1470.726447] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1480.617076] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1487.662267] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1492.579578] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1506.143849] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 0h53m10s [ 1508.074987] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1511.906829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.928294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1511.955734] Lustre: Skipped 1 previous similar message [ 1511.984257] Lustre: Skipped 2 previous similar messages [ 1513.065141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1513.086633] Lustre: Skipped 2 previous similar messages [ 1514.490121] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 1519.588029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.597412] Lustre: Skipped 3 previous similar messages [ 1525.215636] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1525.334645] Lustre: server umount lustre-MDT0000 complete [ 1525.733825] LustreError: 50364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.750572] LustreError: 50364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1527.760990] LustreError: 50359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.588614] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1531.873175] LustreError: 50365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.720544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.878353] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1535.175639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.252398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1536.179685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1539.007905] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1542.719982] Lustre: DEBUG MARKER: Wait for df (36708) ... [ 1544.287869] Lustre: DEBUG MARKER: done [ 1550.816753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1550.833161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.865113] Lustre: Skipped 1 previous similar message [ 1550.891396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1550.898701] Lustre: Skipped 1 previous similar message [ 1551.847847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.595544] Lustre: server umount lustre-OST0000 complete [ 1562.083424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1562.085816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.092763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.092770] Lustre: Skipped 2 previous similar messages [ 1562.116352] Lustre: Skipped 1 previous similar message [ 1565.949062] Lustre: server umount lustre-MDT0000 complete [ 1569.256154] LustreError: 52854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.279044] LustreError: 52854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1570.140691] LustreError: 51098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843257 with bad export cookie 7391243998614192608 [ 1570.151099] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1570.156294] LustreError: 51098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1570.850613] Lustre: server umount lustre-MDT0001 complete [ 1582.907642] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1586.161166] Key type lgssc unregistered [ 1586.468985] LNet: 55289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.474057] LNetError: 55289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.493516] LNet: Removed LNI 192.168.202.117@tcp [ 1587.339055] Key type .llcrypt unregistered [ 1587.344670] Key type ._llcrypt unregistered [ 1630.411210] Key type ._llcrypt registered [ 1630.413968] Key type .llcrypt registered [ 1630.583308] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1632.196545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1632.259213] alg: No test for adler32 (adler32-zlib) [ 1633.361642] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 1633.657535] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1635.375285] Key type lgssc registered [ 1636.841820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1646.442496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1646.484450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.845139] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1647.869866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1648.116035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1653.667188] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1662.338111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.453093] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1662.501344] Lustre: 57496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1662.812522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1669.827740] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1678.996291] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1685.458789] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1693.289662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1693.490931] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1693.746843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.205296] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1703.598376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1709.110862] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1716.688953] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 0h56m43s [ 1718.609254] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1726.778744] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1730.297655] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1730.418771] LustreError: 57504:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1750.603888] Lustre: lustre-MDT0000: Client a5c05015-b765-4f50-bddf-b2213c8475a3 (at 192.168.202.17@tcp) reconnecting [ 1770.431139] LustreError: 57504:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1772.552604] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1774.937433] Lustre: DEBUG MARKER: done [ 1780.194853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.213231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.240251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.715410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.724099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.753537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.208639] Lustre: server umount lustre-OST0000 complete [ 1785.824406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.828752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.847709] Lustre: Skipped 1 previous similar message [ 1785.878623] Lustre: Skipped 1 previous similar message [ 1790.831887] Lustre: server umount lustre-MDT0000 complete [ 1795.606310] LustreError: 57488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843483 with bad export cookie 17363006165709214195 [ 1795.609193] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1795.624340] LustreError: 57488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1795.934307] Lustre: server umount lustre-MDT0001 complete [ 1808.214737] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 1811.713179] Key type lgssc unregistered [ 1812.110821] LNet: 61829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.117469] LNetError: 61829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.155153] LNet: Removed LNI 192.168.202.117@tcp [ 1813.386986] Key type .llcrypt unregistered [ 1813.389618] Key type ._llcrypt unregistered [ 1863.808237] Key type ._llcrypt registered [ 1863.810476] Key type .llcrypt registered [ 1863.933308] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 1865.862104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.962839] alg: No test for adler32 (adler32-zlib) [ 1867.226447] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 1867.581321] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 1869.399181] Key type lgssc registered [ 1870.769021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1875.180771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1875.196620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.562333] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1876.612566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1876.870814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.995506] Lustre: Failing over lustre-MDT0000 [ 1878.370079] Lustre: server umount lustre-MDT0000 complete [ 1885.812521] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:59:32 (1788843572) [ 1893.145696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.553785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.840955] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1907.601655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.678982] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1907.699793] Lustre: 64624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1907.960290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1911.651207] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1918.808844] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1925.746988] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1933.145777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1933.394977] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1933.616397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.655377] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 1942.167737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1948.554854] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1953.886433] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1962.978640] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.980804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.990640] Lustre: Skipped 1 previous similar message [ 1966.885500] Lustre: server umount lustre-MDT0000 complete [ 1970.758857] LustreError: 64617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843658 with bad export cookie 8288581810714605791 [ 1970.764077] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1970.768431] LustreError: 64617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1971.173726] Lustre: server umount lustre-MDT0001 complete [ 1975.459606] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1979.647553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1989.344108] Lustre: 63547:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843661/real 1788843661] req@0000000027760670 x1875738419567872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1989.376147] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.391154] Lustre: Skipped 1 previous similar message [ 1990.523134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.647229] Lustre: 63545:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843663/real 1788843663] req@000000009c7a1a2e x1875738419568000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1994.719357] Lustre: 63544:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843666/real 1788843666] req@00000000d77d5f7d x1875738419568128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1995.583553] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7306f53df76881c2 [ 1995.613369] Lustre: MGC192.168.202.117@tcp: Connection restored to 0@lo (at 0@lo) [ 1996.071059] LustreError: 66471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.096170] LustreError: 66471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1996.273200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.488971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1997.153791] Lustre: 63546:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843668/real 1788843668] req@000000005b5cf4c8 x1875738419568384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1998.240183] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.292081] LustreError: 66471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.337481] LustreError: 66471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2001.312270] Lustre: 63546:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843673/real 1788843673] req@000000004e7cfeb7 x1875738419568512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.802207] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2004.469790] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.499663] LustreError: 66472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2009.568957] LustreError: 68407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.576893] LustreError: 68407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2011.001443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.226686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2015.950225] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2016.249276] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2018.353101] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2023.887962] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2031.468730] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2041.826647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2041.853340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.893742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.903926] Lustre: Skipped 2 previous similar messages [ 2043.872755] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.875929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.887569] Lustre: Skipped 1 previous similar message [ 2043.900101] Lustre: Skipped 2 previous similar messages [ 2045.249805] Lustre: server umount lustre-MDT0000 complete [ 2048.997641] LustreError: 68543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.016911] LustreError: 68543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2049.800214] LustreError: 64615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843737 with bad export cookie 8288581810714608066 [ 2049.808959] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2049.815968] LustreError: 64615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2050.141090] Lustre: server umount lustre-MDT0001 complete [ 2056.946603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2067.178105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.473038] Lustre: 63547:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843741/real 1788843741] req@000000004c6ee82e x1875738419598336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.499082] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.655641] LustreError: 63543:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000455cb026 x1875738419599232/t0(0) o250->MGC192.168.202.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2075.120045] LustreError: 69142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.197575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.276681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2080.375526] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2089.884662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.600812] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2095.962615] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2104.157565] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2110.642061] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2119.137835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.156584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.644953] Lustre: server umount lustre-MDT0000 complete [ 2124.265641] LustreError: 71089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.281823] LustreError: 71089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2127.335389] LustreError: 64617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843815 with bad export cookie 8288581810714609445 [ 2127.339555] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2127.341952] LustreError: 64617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2127.602750] Lustre: server umount lustre-MDT0001 complete [ 2133.826720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2137.351847] Lustre: DEBUG MARKER: files compared the same [ 2149.783101] Lustre: server umount lustre-OST0000 complete [ 2168.804775] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2171.712275] Key type lgssc unregistered [ 2172.099122] LNet: 74366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.109711] LNetError: 74366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.130318] LNet: Removed LNI 192.168.202.117@tcp [ 2173.073676] Key type .llcrypt unregistered [ 2173.079909] Key type ._llcrypt unregistered [ 2198.642758] Key type ._llcrypt registered [ 2198.644678] Key type .llcrypt registered [ 2198.773485] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2200.476673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2200.550527] alg: No test for adler32 (adler32-zlib) [ 2201.639486] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 2202.002776] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2203.703193] Key type lgssc registered [ 2205.289419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2213.933404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2214.008607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.480645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.547223] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug +malloc all [ 2225.639460] LustreError: 75781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.657592] LustreError: 75781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2229.329279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.644517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2234.578342] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug +malloc all [ 2241.330073] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2246.746935] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2254.510595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2255.247606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.416732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2263.229925] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug +malloc all [ 2272.806659] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2285.025047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.025172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2285.057298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2285.072123] Lustre: Skipped 1 previous similar message [ 2289.718759] Lustre: server umount lustre-OST0000 complete [ 2295.279670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.305093] Lustre: Skipped 1 previous similar message [ 2295.308776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.330346] Lustre: Skipped 1 previous similar message [ 2299.853428] Lustre: server umount lustre-MDT0000 complete [ 2300.387221] LustreError: 76509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.418426] LustreError: 76509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2304.146089] LustreError: 75760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843991 with bad export cookie 702510643378420581 [ 2304.157117] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.158838] LustreError: 75760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2304.499374] Lustre: server umount lustre-MDT0001 complete [ 2317.793500] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2321.589793] Key type lgssc unregistered [ 2322.081498] LNet: 79318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.102708] LNetError: 79318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2322.144024] LNet: Removed LNI 192.168.202.117@tcp [ 2323.712777] Key type .llcrypt unregistered [ 2323.723248] Key type ._llcrypt unregistered [ 2352.610514] Key type ._llcrypt registered [ 2352.613420] Key type .llcrypt registered [ 2352.726852] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2353.703711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2353.883105] alg: No test for adler32 (adler32-zlib) [ 2355.094581] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 2355.371386] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2357.039450] Key type lgssc registered [ 2358.241772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2366.293830] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2366.312622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2381.855588] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2386.975401] LustreError: 80708:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.117@tcp: failed processing log, type 4: rc = -110 [ 2413.663238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.137683] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2428.745804] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2436.243135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.388744] Lustre: *** cfs_fail_loc=706, val=0*** [ 2436.922567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.052791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2441.406708] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2442.216767] LustreError: 80733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.238734] LustreError: 80733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2447.341678] LustreError: 80734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.420713] LustreError: 81791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.450670] LustreError: 81791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2450.414556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.747690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2456.418848] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2464.288105] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2470.285234] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2472.928426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2472.931235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.952572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.514176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.523773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.532700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2478.578586] Lustre: server umount lustre-OST0000 complete [ 2486.759519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.764333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2486.766765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.766773] Lustre: Skipped 1 previous similar message [ 2486.771134] Lustre: Skipped 1 previous similar message [ 2488.069297] Lustre: server umount lustre-MDT0000 complete [ 2491.641463] LustreError: 80716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844179 with bad export cookie 7709882408627560156 [ 2491.668150] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2492.008531] Lustre: server umount lustre-MDT0001 complete [ 2501.654630] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2504.738864] Key type lgssc unregistered [ 2505.028883] LNet: 84322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2505.037473] LNetError: 84322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2505.058569] LNet: Removed LNI 192.168.202.117@tcp [ 2505.881927] Key type .llcrypt unregistered [ 2505.890592] Key type ._llcrypt unregistered [ 2529.910996] Key type ._llcrypt registered [ 2529.914311] Key type .llcrypt registered [ 2530.046438] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2531.387024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2531.446752] alg: No test for adler32 (adler32-zlib) [ 2532.618952] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 2533.030968] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2534.755353] Key type lgssc registered [ 2535.820682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2542.698756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2542.720133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.644591] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2555.750243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.232397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2560.965789] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2561.504496] LustreError: 86421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.526560] LustreError: 86421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2565.614045] LustreError: 86424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.630658] LustreError: 86424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2569.305612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.637450] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.656054] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2569.670823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.846741] LustreError: 86419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.854939] LustreError: 86419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2576.039628] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2579.938557] LustreError: 86424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.963082] LustreError: 86424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2584.916423] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.936732] Lustre: MGS: Client 5ae71747-5abe-4c08-80dc-381b697ec78c (at 0@lo) reconnecting [ 2584.955735] LustreError: 86435:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2585.244788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.321665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2589.927029] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2606.052953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.086169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2606.094724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2608.804779] Lustre: server umount lustre-OST0000 complete [ 2616.289514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.289891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2616.297320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.297330] Lustre: Skipped 1 previous similar message [ 2616.299380] Lustre: Skipped 2 previous similar messages [ 2619.151401] Lustre: server umount lustre-MDT0000 complete [ 2621.411145] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.441033] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2623.264149] Lustre: server umount lustre-MDT0001 complete [ 2627.259869] Lustre: server umount MGS complete [ 2641.387902] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:12:07 (1788844327) [ 2670.528505] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 2680.804286] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2715.802399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.191317] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2729.422936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.630288] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2729.717935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2729.815457] Lustre: lustre-MDT0001: new disk, initializing [ 2729.898647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2729.959918] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2729.977066] Lustre: 93619:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2729.989671] LustreError: 93619:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2734.091450] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2744.942912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2745.201934] Lustre: lustre-OST0000: new disk, initializing [ 2745.205961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2745.212401] Lustre: 94385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2745.281343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.288829] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2753.634507] Lustre: 94754:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000020077f27 x1875739116871552/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788844457 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2761.757824] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2768.059154] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.076309] Lustre: MGS: Client 58ca164e-e119-461d-a7ba-d395a1f8ed24 (at 0@lo) reconnecting [ 2768.105743] Lustre: 92886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2768.117978] LustreError: 93605:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2768.150840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2768.223903] Lustre: lustre-MDT0000: new disk, initializing [ 2768.413543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.446363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2768.467520] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2769.504849] Lustre: 95454:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000cfdd71fd x1875739116878592/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788844473 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2773.417085] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2774.624202] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2778.606729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2778.633545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2778.698401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2804.705844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2804.714837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.726513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2804.733155] Lustre: Skipped 1 previous similar message [ 2809.728172] Lustre: server umount lustre-OST0000 complete [ 2816.494214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2816.499824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.505505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2816.505512] Lustre: Skipped 2 previous similar messages [ 2816.538976] Lustre: Skipped 2 previous similar messages [ 2821.406490] Lustre: server umount lustre-MDT0000 complete [ 2821.601366] LustreError: 93591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.609537] LustreError: 93591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2831.891854] Lustre: server umount lustre-MDT0001 complete [ 2836.324294] Lustre: server umount MGS complete [ 2847.259613] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:15:34 (1788844534) [ 2865.007472] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 2867.381562] Key type lgssc unregistered [ 2867.683781] LNet: 97952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2867.690215] LNetError: 97952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2867.715769] LNet: Removed LNI 192.168.202.117@tcp [ 2868.856289] Key type .llcrypt unregistered [ 2868.859963] Key type ._llcrypt unregistered [ 2889.230409] Key type ._llcrypt registered [ 2889.233613] Key type .llcrypt registered [ 2889.385567] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 2890.957509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2891.089533] alg: No test for adler32 (adler32-zlib) [ 2892.120778] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 2892.442260] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 2894.135576] Key type lgssc registered [ 2895.830085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2904.674515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2904.690768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.174174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.528967] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2915.300194] LustreError: 99173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2915.310166] LustreError: 99173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2918.002621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.307465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.912519] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2931.680563] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2938.199598] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2947.039344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.655233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.711585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2955.134245] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 2964.405174] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2974.185955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.189559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.194114] Lustre: Skipped 1 previous similar message [ 2979.298376] Lustre: server umount lustre-OST0000 complete [ 2984.425141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.434128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2984.453393] Lustre: Skipped 1 previous similar message [ 2984.458701] Lustre: Skipped 1 previous similar message [ 2989.540460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2989.555940] Lustre: Skipped 2 previous similar messages [ 2990.539355] Lustre: server umount lustre-MDT0000 complete [ 2994.664858] LustreError: 101770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2994.694520] LustreError: 101770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2995.049218] LustreError: 99155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844682 with bad export cookie 9631397828881803759 [ 2995.063595] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2995.073502] LustreError: 99155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2995.785344] Lustre: server umount lustre-MDT0001 complete [ 3009.873686] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3013.620923] Key type lgssc unregistered [ 3013.961607] LNet: 102706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3013.970278] LNetError: 102706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3014.000086] LNet: Removed LNI 192.168.202.117@tcp [ 3015.149132] Key type .llcrypt unregistered [ 3015.153246] Key type ._llcrypt unregistered [ 3033.251051] Key type ._llcrypt registered [ 3033.253987] Key type .llcrypt registered [ 3033.360620] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3034.547305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3034.713117] alg: No test for adler32 (adler32-zlib) [ 3035.884797] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 3036.231556] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3037.887876] Key type lgssc registered [ 3039.144160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3045.150978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3045.165905] LustreError: 103774:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3045.212702] LustreError: 103786:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3045.220361] LustreError: 103774:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4952 [ 3045.228881] LustreError: 103774:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3045.241482] LustreError: 103774:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3045.248845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.868115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.393166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.623486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3061.074475] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3066.821761] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing lsmod [ 3070.013039] LustreError: 104990:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3070.043245] LustreError: 104994:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3070.049049] LustreError: 104990:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 3070.055633] LustreError: 104990:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3070.064271] LustreError: 104990:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3070.069100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.428566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.483923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3076.065325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3076.071502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.085780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.584123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.590227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.608196] Lustre: Skipped 1 previous similar message [ 3080.737723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.742442] Lustre: Skipped 3 previous similar messages [ 3085.793627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3085.806078] Lustre: Skipped 2 previous similar messages [ 3088.351237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3088.810826] Lustre: server umount lustre-MDT0000 complete [ 3090.914151] LustreError: 103820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.934982] LustreError: 103820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3093.099977] LustreError: 103800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844780 with bad export cookie 16337077224900936521 [ 3093.121569] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3095.135423] LustreError: 104256:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009fddd145 x1875739645336064/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3095.352128] Lustre: server umount lustre-MDT0001 complete [ 3108.494293] Lustre: server umount lustre-OST0000 complete [ 3117.455662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.820330] LustreError: 106060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3117.911303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.171845] LustreError: 106061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3123.326832] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3127.266897] LustreError: 106060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.386888] LustreError: 106061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.413098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.830726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3137.629664] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3146.812613] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3152.811682] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3159.809562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3160.121854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.186359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3166.325310] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3173.821559] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3186.151502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3186.160517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.179119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3187.680257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.575764] Lustre: server umount lustre-OST0000 complete [ 3194.336706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3194.341572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.363689] Lustre: Skipped 1 previous similar message [ 3194.373275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.387793] Lustre: Skipped 2 previous similar messages [ 3200.760567] Lustre: server umount lustre-MDT0000 complete [ 3203.047796] LustreError: 106061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.077871] LustreError: 106061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3205.901125] LustreError: 106042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844893 with bad export cookie 16337077224900937438 [ 3205.919671] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3205.925468] LustreError: 106042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3206.292053] Lustre: server umount lustre-MDT0001 complete [ 3219.371417] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3221.955561] Key type lgssc unregistered [ 3222.317730] LNet: 109593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3222.334529] LNetError: 109593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3222.367024] LNet: Removed LNI 192.168.202.117@tcp [ 3223.335816] Key type .llcrypt unregistered [ 3223.337944] Key type ._llcrypt unregistered [ 3252.110940] Key type ._llcrypt registered [ 3252.113774] Key type .llcrypt registered [ 3252.267117] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3253.608634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3253.736354] alg: No test for adler32 (adler32-zlib) [ 3254.946834] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 3255.297891] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3257.039534] Key type lgssc registered [ 3258.590387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3268.404608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3268.428931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.118308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3275.466320] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3279.330589] LustreError: 111008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.368649] LustreError: 111008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3284.463494] LustreError: 111009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.865671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.269485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3290.101668] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3298.128860] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3304.804453] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3312.357343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3312.850614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.947752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3320.480291] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3328.829255] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3334.529594] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3337.405826] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3340.777639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3340.792152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.818851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.660598] Lustre: server umount lustre-OST0000 complete [ 3349.987341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.988162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3350.008835] Lustre: Skipped 1 previous similar message [ 3350.049757] Lustre: Skipped 1 previous similar message [ 3353.504771] Lustre: server umount lustre-MDT0000 complete [ 3355.127048] LustreError: 111009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.173246] LustreError: 111009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3357.449432] LustreError: 110990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845045 with bad export cookie 17522753188327857759 [ 3357.455074] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.459949] LustreError: 110990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3368.297403] Lustre: server umount lustre-MDT0001 complete [ 3379.427147] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3381.810790] Key type lgssc unregistered [ 3382.228110] LNet: 114642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3382.239550] LNetError: 114642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3382.258402] LNet: Removed LNI 192.168.202.117@tcp [ 3383.212218] Key type .llcrypt unregistered [ 3383.214562] Key type ._llcrypt unregistered [ 3407.540495] Key type ._llcrypt registered [ 3407.546243] Key type .llcrypt registered [ 3407.717939] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3409.140394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3409.214037] alg: No test for adler32 (adler32-zlib) [ 3411.055338] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 3411.670389] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3413.399171] Key type lgssc registered [ 3415.597719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3425.982316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3426.024097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.578792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.286365] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3438.056205] LustreError: 115862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.082337] LustreError: 115862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3441.778130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.291950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3447.777217] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3455.670704] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3461.940983] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3469.003672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3469.484294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.568317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3476.090197] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3484.733135] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3496.932051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3496.940536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3496.955091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3497.446684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3497.462314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3497.475825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3497.480777] Lustre: Skipped 1 previous similar message [ 3501.087076] Lustre: server umount lustre-OST0000 complete [ 3507.695452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.723473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3507.726180] Lustre: Skipped 1 previous similar message [ 3510.605723] Lustre: server umount lustre-MDT0000 complete [ 3512.800740] LustreError: 115857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.823933] LustreError: 115857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3513.937191] LustreError: 115843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845201 with bad export cookie 7180675254797736119 [ 3513.961786] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3514.248859] Lustre: server umount lustre-MDT0001 complete [ 3524.956925] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3527.850163] Key type lgssc unregistered [ 3528.125060] LNet: 119394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3528.130882] LNetError: 119394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3528.149305] LNet: Removed LNI 192.168.202.117@tcp [ 3529.009215] Key type .llcrypt unregistered [ 3529.011244] Key type ._llcrypt unregistered [ 3554.090232] Key type ._llcrypt registered [ 3554.093645] Key type .llcrypt registered [ 3554.321872] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3556.027748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3556.070925] alg: No test for adler32 (adler32-zlib) [ 3557.391443] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 3557.706833] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3559.415162] Key type lgssc registered [ 3561.543524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3574.852575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3574.878884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.509088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.101073] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3587.048089] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.070357] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3589.720910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.987308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3594.425410] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3603.975288] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3610.312574] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3617.344594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3617.624970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3621.774033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3624.904735] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3634.219615] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3641.549418] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3645.364572] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3645.372760] Lustre: Skipped 2 previous similar messages [ 3647.053569] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3647.096825] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3649.266778] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3649.273311] Lustre: Skipped 2 previous similar messages [ 3654.226584] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3654.238226] Lustre: Skipped 1 previous similar message [ 3660.140860] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.17@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3660.160164] Lustre: Skipped 2 previous similar messages [ 3669.562853] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.17@tcp 192.168.0.[2,10]@tcp [ 3675.336944] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.17/24@tcp in log lustre-MDT0000 [ 3675.347021] Lustre: Skipped 2 previous similar messages [ 3676.801243] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.17/24@tcp [ 3676.810033] Lustre: Skipped 1 previous similar message [ 3687.911929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3687.916516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.925750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3691.488109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3691.490157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3691.499380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.511151] Lustre: Skipped 1 previous similar message [ 3694.281192] Lustre: server umount lustre-OST0000 complete [ 3701.746269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.773221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3701.791616] Lustre: Skipped 1 previous similar message [ 3704.445655] Lustre: server umount lustre-MDT0000 complete [ 3706.849613] LustreError: 121821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3706.897335] LustreError: 121821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3708.625545] LustreError: 123439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845396 with bad export cookie 10303923860159723955 [ 3708.635174] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3708.644580] LustreError: 123439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3709.254844] Lustre: server umount lustre-MDT0001 complete [ 3723.191388] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3727.288727] Key type lgssc unregistered [ 3727.660900] LNet: 125750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3727.665673] LNetError: 125750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3727.677487] LNet: Removed LNI 192.168.202.117@tcp [ 3728.269581] Key type .llcrypt unregistered [ 3728.288320] Key type ._llcrypt unregistered [ 3757.443601] Key type ._llcrypt registered [ 3757.445946] Key type .llcrypt registered [ 3757.610391] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3758.639057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3758.665102] alg: No test for adler32 (adler32-zlib) [ 3759.712964] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 3759.972218] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3761.672468] Key type lgssc registered [ 3762.875216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3769.748768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3769.764345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3771.251391] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3771.299860] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.17/24@tcp [ 3771.355167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.869466] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3781.600097] LustreError: 127308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.637645] LustreError: 127308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3786.732690] LustreError: 127309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.888225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.341981] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3788.378544] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.17/24@tcp [ 3788.422033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3792.923195] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3800.503616] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3806.878132] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3814.117575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3814.606514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3816.667700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3820.303741] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3827.718309] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3842.027811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3842.037917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.048650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.588129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.600680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3844.605565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.698046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3849.702879] Lustre: Skipped 1 previous similar message [ 3854.828993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3854.846239] Lustre: Skipped 2 previous similar messages [ 3856.351144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3856.437829] Lustre: server umount lustre-OST0000 complete [ 3865.058526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3865.060508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.073397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3865.079421] Lustre: Skipped 1 previous similar message [ 3865.091489] Lustre: Skipped 1 previous similar message [ 3866.678487] Lustre: server umount lustre-MDT0000 complete [ 3870.176883] LustreError: 128885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3870.196531] LustreError: 128885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3870.926385] LustreError: 127288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845558 with bad export cookie 16242998613512190041 [ 3870.944014] LustreError: 127288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3870.945631] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3871.808169] Lustre: server umount lustre-MDT0001 complete [ 3885.488796] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 3889.774243] Key type lgssc unregistered [ 3890.122503] LNet: 130902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3890.132585] LNetError: 130902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3891.186815] LNet: Removed LNI 192.168.202.117@tcp [ 3892.663107] Key type .llcrypt unregistered [ 3892.665050] Key type ._llcrypt unregistered [ 3922.436541] Key type ._llcrypt registered [ 3922.438646] Key type .llcrypt registered [ 3922.537587] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 3938.200861] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 3939.264153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3939.327133] alg: No test for adler32 (adler32-zlib) [ 3940.366548] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 3940.566981] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 3942.239411] Key type lgssc registered [ 3943.620220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3979.559418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3979.568450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.862837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3980.904409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3980.991290] Lustre: lustre-MDT0000: new disk, initializing [ 3981.082039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3981.109123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3985.709931] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 3996.103624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.259638] Lustre: 134300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3996.336145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3996.340495] Lustre: Skipped 1 previous similar message [ 3996.455810] Lustre: lustre-MDT0001: new disk, initializing [ 3996.591702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3996.657117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3996.682688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4001.656875] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4011.403334] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4016.761405] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4023.589433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4023.809083] Lustre: lustre-OST0000: new disk, initializing [ 4023.810854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4023.813672] Lustre: 136268:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4023.861178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.026623] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4032.588663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4032.599110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4032.667912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4040.665208] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4046.606160] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4046.831870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4053.110456] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4053.385916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4058.609958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.633567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4058.639769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4058.676043] LustreError: Skipped 1 previous similar message [ 4061.994097] Lustre: server umount lustre-OST0000 complete [ 4068.329874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4068.351938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.379288] Lustre: Skipped 1 previous similar message [ 4068.383695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4068.388430] Lustre: Skipped 1 previous similar message [ 4072.294676] Lustre: server umount lustre-MDT0000 complete [ 4076.240310] LustreError: 134291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845764 with bad export cookie 4146007007932302402 [ 4076.244187] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4076.258827] LustreError: 134291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4076.701431] Lustre: server umount lustre-MDT0001 complete [ 4086.186429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.610395] LustreError: 138485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.621331] LustreError: 138485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4086.722960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.591459] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4091.877130] LustreError: 138486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.998160] LustreError: 138485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4100.552219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.010091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4106.041784] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4114.388574] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4120.398782] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4127.271268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4127.512836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.226519] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4139.442546] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4144.299171] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4144.556442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4149.222590] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4149.402427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4157.383787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4157.553310] Lustre: lustre-OST0001: new disk, initializing [ 4157.566300] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4157.580774] Lustre: 141857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4157.686533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4163.734106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4163.749864] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4163.874995] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4164.280811] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4174.120930] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4174.433977] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4180.486324] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4180.771193] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4187.435280] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff971d87726800.ost_server_uuid 50 [ 4189.051698] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff971d87726800.ost_server_uuid in IDLE state after 0 sec [ 4199.904637] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4199.921170] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.921622] LustreError: Skipped 1 previous similar message [ 4199.925139] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4199.940921] Lustre: Skipped 2 previous similar messages [ 4199.950101] Lustre: Skipped 1 previous similar message [ 4200.334201] Lustre: server umount lustre-OST0001 complete [ 4205.027061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.037694] Lustre: Skipped 1 previous similar message [ 4205.038728] LustreError: 140307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.039423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4205.039430] Lustre: Skipped 1 previous similar message [ 4205.065717] LustreError: 140307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4210.146846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4210.150205] Lustre: Skipped 1 previous similar message [ 4210.158223] LustreError: 141509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4210.184827] LustreError: 141509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4219.361448] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4219.492963] Lustre: server umount lustre-OST0000 complete [ 4223.970240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4223.978691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.992666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4223.996228] Lustre: Skipped 2 previous similar messages [ 4238.303136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4238.523448] Lustre: server umount lustre-MDT0000 complete [ 4240.865510] LustreError: 138486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.874714] LustreError: 138486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4242.020600] LustreError: 141856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845929 with bad export cookie 4146007007932303284 [ 4242.025067] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.026456] LustreError: 141856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4242.323965] Lustre: server umount lustre-MDT0001 complete [ 4260.492252] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 4263.273089] Key type lgssc unregistered [ 4263.721766] LNet: 144860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4263.735712] LNetError: 144860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4263.761308] LNet: Removed LNI 192.168.202.117@tcp [ 4264.598732] Key type .llcrypt unregistered [ 4264.600790] Key type ._llcrypt unregistered [ 4302.932501] Key type ._llcrypt registered [ 4302.936622] Key type .llcrypt registered [ 4303.118651] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 4321.665822] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4322.751505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4322.828567] alg: No test for adler32 (adler32-zlib) [ 4324.114687] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 4324.514780] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 4326.247373] Key type lgssc registered [ 4327.858220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4363.407415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4363.442988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.806065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4364.839196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4364.963662] Lustre: lustre-MDT0000: new disk, initializing [ 4365.064478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.084684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4369.612644] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4380.673040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4380.853472] Lustre: 149009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4380.954402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4380.964249] Lustre: Skipped 1 previous similar message [ 4381.093172] Lustre: lustre-MDT0001: new disk, initializing [ 4381.241861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4381.301932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4381.313701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4386.522824] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4396.283189] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4402.486162] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4409.935546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4410.404750] Lustre: lustre-OST0000: new disk, initializing [ 4410.410730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4410.418531] Lustre: 150983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4410.529238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.300669] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4417.639863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4417.657643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4417.745746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4425.126852] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4429.746739] Lustre: Failing over lustre-OST0000 [ 4429.907632] Lustre: server umount lustre-OST0000 complete [ 4447.711969] Lustre: 147236:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788846119/real 1788846119] req@000000003e268696 x1875740995912192/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788846135 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4447.745178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.299437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4449.782153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.811835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.835257] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4450.848641] Lustre: 147235:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788846121/real 1788846121] req@00000000e544d1d0 x1875740995912576/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788846137 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4452.840660] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4454.879908] Lustre: 147237:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788846126/real 1788846126] req@00000000f77cb764 x1875740995914368/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788846142 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4454.940194] Lustre: 147237:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4457.662423] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4459.562536] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4459.566877] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 4460.067094] Lustre: 147234:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788846131/real 1788846131] req@00000000791debe2 x1875740995915520/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788846147 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4461.962763] Lustre: Failing over lustre-MDT0000 [ 4462.354075] Lustre: server umount lustre-MDT0000 complete [ 4463.075213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4463.093519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.112441] Lustre: Skipped 1 previous similar message [ 4464.167446] Lustre: 147236:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788846135/real 1788846135] req@000000007ed7be35 x1875740995916416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788846151 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4464.193692] Lustre: 147236:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4468.193269] LustreError: 149023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.218260] LustreError: 149023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4469.728684] LustreError: 149019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.340123] LustreError: 149017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.357564] LustreError: 149017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4478.434773] LustreError: 149022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.453082] LustreError: 149022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4479.456576] Lustre: 147237:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788846150/real 1788846150] req@000000003e268696 x1875740995926272/t0(0) o400->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788846166 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4479.476595] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4480.997626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.052383] LustreError: 149019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.076767] LustreError: 149019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4490.022138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.080965] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4490.191442] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4495.235442] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4495.366768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4495.383429] Lustre: Skipped 1 previous similar message [ 4495.418903] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4495.493434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4505.569897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4505.585338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.588236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4505.600484] Lustre: Skipped 1 previous similar message [ 4510.561072] Lustre: server umount lustre-OST0000 complete [ 4515.809520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4515.814803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.823869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4515.854829] Lustre: Skipped 1 previous similar message [ 4515.867470] Lustre: Skipped 2 previous similar messages [ 4520.687379] Lustre: server umount lustre-MDT0000 complete [ 4520.933502] LustreError: 150194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.948647] LustreError: 150194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4524.090479] LustreError: 152783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846211 with bad export cookie 8718520722845440021 [ 4524.106982] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4524.110434] LustreError: 152783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4524.495827] Lustre: server umount lustre-MDT0001 complete [ 4534.201038] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 4536.548694] Key type lgssc unregistered [ 4536.879949] LNet: 154640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4536.887926] LNetError: 154640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4536.922315] LNet: Removed LNI 192.168.202.117@tcp [ 4538.011972] Key type .llcrypt unregistered [ 4538.014028] Key type ._llcrypt unregistered [ 4574.237951] Key type ._llcrypt registered [ 4574.239892] Key type .llcrypt registered [ 4574.330396] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4575.287045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4575.305496] alg: No test for adler32 (adler32-zlib) [ 4576.486940] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 4576.843794] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 4578.503193] Key type lgssc registered [ 4579.470669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4586.325090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4586.347237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.720536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4592.390494] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4597.219400] LustreError: 156160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.251698] LustreError: 156160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4602.338035] LustreError: 156159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4603.931762] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4607.461316] LustreError: 156159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4608.323764] LustreError: 156188:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4608.343562] LustreError: 156188:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4608.358137] LustreError: 156188:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4608.885911] Lustre: server umount lustre-MDT0000 complete [ 4624.146521] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 4633.552748] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 4670.936657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.171277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4671.208556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4671.349648] Lustre: lustre-MDT0000: new disk, initializing [ 4671.460508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.489691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4676.421541] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4686.728224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.959470] Lustre: 159488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4687.063335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4687.066244] Lustre: Skipped 1 previous similar message [ 4687.161291] Lustre: lustre-MDT0001: new disk, initializing [ 4687.300552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4687.356809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4687.373907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4692.130733] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4700.644506] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4706.270640] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4712.458452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4712.665744] Lustre: lustre-OST0000: new disk, initializing [ 4712.669213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4712.675492] Lustre: 161457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4712.760272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4718.760687] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 4721.809838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4721.821729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4721.992988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4729.333871] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4734.585185] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4737.231578] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:47:03 (1788846423) [ 4957.664192] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.673055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4957.686246] Lustre: Skipped 1 previous similar message [ 4957.700165] Lustre: Skipped 2 previous similar messages [ 4962.790963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4962.796664] Lustre: Skipped 2 previous similar messages [ 4963.602857] Lustre: server umount lustre-MDT0000 complete [ 4967.484785] LustreError: 159481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846655 with bad export cookie 6566835773514149034 [ 4967.486099] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.493406] LustreError: 159481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4967.919621] LustreError: 159496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.920036] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4967.945261] Lustre: Skipped 1 previous similar message [ 4967.953537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4973.027532] LustreError: 163234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.034415] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4973.936936] Lustre: server umount lustre-MDT0001 complete [ 4984.730538] Lustre: server umount lustre-OST0000 complete [ 4995.315684] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 5006.000639] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5043.242657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.497493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5043.527227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5043.605434] Lustre: lustre-MDT0000: new disk, initializing [ 5043.680586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5043.704771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5048.924536] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5059.784406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.913779] Lustre: 166118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5059.981143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5059.987735] Lustre: Skipped 1 previous similar message [ 5060.049937] Lustre: lustre-MDT0001: new disk, initializing [ 5060.169884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5060.215889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5060.224730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5065.101520] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5074.229749] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5079.761658] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5086.207325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5086.516884] Lustre: lustre-OST0000: new disk, initializing [ 5086.522983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5086.541079] Lustre: 168085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5086.641378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5093.656157] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5096.565936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5096.588265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5096.734137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5103.782809] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5108.804457] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5109.034251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5114.337773] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5114.488878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5117.415404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5117.424023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5117.435042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5117.437125] LustreError: Skipped 1 previous similar message [ 5117.479452] Lustre: Skipped 1 previous similar message [ 5122.422960] Lustre: server umount lustre-OST0000 complete [ 5126.628784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5126.644736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5126.662889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.665575] Lustre: Skipped 1 previous similar message [ 5132.627817] Lustre: server umount lustre-MDT0000 complete [ 5132.772556] LustreError: 167283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.794759] LustreError: 167283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5137.135239] LustreError: 166110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846824 with bad export cookie 6566835773514307864 [ 5137.158636] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5137.160220] LustreError: 166110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5137.732907] Lustre: server umount lustre-MDT0001 complete [ 5147.549467] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:53:53 (1788846833) [ 5165.641985] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 5175.409072] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5215.813389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.169847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5216.193211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5216.277933] Lustre: lustre-MDT0000: new disk, initializing [ 5216.363685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5216.374681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5222.116956] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5233.440832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.544378] Lustre: 172842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5233.755585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5238.015271] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5247.812772] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5254.408939] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5262.177190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5262.585127] Lustre: lustre-OST0000: new disk, initializing [ 5262.591623] Lustre: Skipped 1 previous similar message [ 5262.607877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5262.621309] Lustre: Skipped 2 previous similar messages [ 5262.631538] Lustre: 174824:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5271.077493] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5271.156330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5271.167465] Lustre: Skipped 1 previous similar message [ 5271.173261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5271.327674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5281.930506] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5292.002300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5292.006259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.036857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5292.048774] Lustre: Skipped 2 previous similar messages [ 5292.064084] Lustre: Skipped 3 previous similar messages [ 5304.287153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5304.508711] Lustre: server umount lustre-OST0000 complete [ 5310.432585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5310.454781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5315.214735] Lustre: server umount lustre-MDT0000 complete [ 5317.600235] LustreError: 172855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5317.621696] LustreError: 172855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5319.525924] LustreError: 173645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847007 with bad export cookie 6566835773514308746 [ 5319.535819] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5319.546994] LustreError: 173645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5320.035219] Lustre: server umount lustre-MDT0001 complete [ 5329.207765] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:56:55 (1788847015) [ 5345.362819] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 5356.613821] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5394.375382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.700463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5394.758812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5394.934369] Lustre: lustre-MDT0000: new disk, initializing [ 5395.061072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.073232] Lustre: Skipped 2 previous similar messages [ 5395.085940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5400.719225] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5413.132387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.324411] Lustre: 179145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5413.707966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5419.038867] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5429.442675] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5435.877933] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5443.428590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5443.738507] Lustre: 181125:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5450.291123] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5452.925719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5453.075256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5459.962129] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5468.654030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.661073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5468.675499] Lustre: Skipped 1 previous similar message [ 5468.679674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5468.705595] Lustre: Skipped 7 previous similar messages [ 5472.880274] Lustre: server umount lustre-OST0000 complete [ 5491.167250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5491.343562] Lustre: server umount lustre-MDT0000 complete [ 5494.241843] LustreError: 180714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5494.268803] LustreError: 180714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5495.402149] LustreError: 179140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847183 with bad export cookie 6566835773514309831 [ 5495.412608] LustreError: 179140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5495.414048] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5495.772977] Lustre: server umount lustre-MDT0001 complete [ 5509.076202] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5512.475624] Key type lgssc unregistered [ 5512.789181] LNet: 182980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5512.794257] LNetError: 182980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5512.808883] LNet: Removed LNI 192.168.202.117@tcp [ 5513.639633] Key type .llcrypt unregistered [ 5513.642916] Key type ._llcrypt unregistered [ 5539.202369] Key type ._llcrypt registered [ 5539.205729] Key type .llcrypt registered [ 5539.319261] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5540.470081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5540.484049] alg: No test for adler32 (adler32-zlib) [ 5541.458110] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 5541.711437] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5543.383971] Key type lgssc registered [ 5544.784713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5553.453538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5553.474550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.034376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.031956] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5564.386930] LustreError: 184394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5564.407375] LustreError: 184394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5567.343633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.772780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5572.632970] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5580.839712] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5587.042365] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5593.168200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5593.439577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.607897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5600.561696] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5608.268220] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5630.447248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5630.465936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.488272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5633.507764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.510048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5633.543788] Lustre: Skipped 1 previous similar message [ 5635.456636] Lustre: server umount lustre-OST0000 complete [ 5639.649432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5639.655827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.668059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5643.745139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5643.746884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5643.780303] Lustre: Skipped 1 previous similar message [ 5645.362795] Lustre: server umount lustre-MDT0000 complete [ 5648.864575] LustreError: 185399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5648.875593] LustreError: 185399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5648.910609] LustreError: 184374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847336 with bad export cookie 13881289207054922320 [ 5648.924756] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5648.930200] LustreError: 184374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5649.504106] Lustre: server umount lustre-MDT0001 complete [ 5661.697812] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 5665.024749] Key type lgssc unregistered [ 5665.521617] LNet: 187961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5665.526687] LNetError: 187961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5665.536868] LNet: Removed LNI 192.168.202.117@tcp [ 5666.534324] Key type .llcrypt unregistered [ 5666.539553] Key type ._llcrypt unregistered [ 5690.297517] Key type ._llcrypt registered [ 5690.300644] Key type .llcrypt registered [ 5690.400876] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 5691.828595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5691.895052] alg: No test for adler32 (adler32-zlib) [ 5693.037232] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 5693.334955] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 5695.088361] Key type lgssc registered [ 5696.115527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5702.670539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5702.701739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5704.093421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.791939] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5713.396824] LustreError: 189374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.421071] LustreError: 189374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5716.558632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.925008] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5720.902326] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5730.554229] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5736.124297] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5742.900294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5743.369533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5748.545568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5750.024217] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5759.645764] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5769.184665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.188892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.195709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.778464] Lustre: lustre-OST0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 5772.259575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5772.284893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.952380] Lustre: server umount lustre-OST0000 complete [ 5783.194904] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff971d99630000.ost_server_uuid 50 [ 5785.085838] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5802.984816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.998205] Lustre: Skipped 1 previous similar message [ 5803.000092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5803.012420] Lustre: Skipped 3 previous similar messages [ 5805.681317] Lustre: server umount lustre-MDT0000 complete [ 5808.098023] LustreError: 190102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5808.111633] LustreError: 190102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5810.015204] LustreError: 189355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847497 with bad export cookie 6813393071078361160 [ 5810.030072] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5810.031312] LustreError: 189355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5810.435403] Lustre: server umount lustre-MDT0001 complete [ 5819.904540] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:05:06 (1788847506) [ 5828.151557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.684987] LustreError: 193290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.812315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.983392] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5834.214427] LustreError: 193291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.311061] LustreError: 193290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.011851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.280820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5847.775446] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5854.918557] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5860.152861] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5867.287733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5867.559951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5873.336653] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5873.768538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5883.021695] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5891.333055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5891.603884] Lustre: lustre-OST0001: new disk, initializing [ 5891.608282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5891.620656] Lustre: 196132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5891.780609] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5898.475175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5898.501768] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5898.710865] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5900.327383] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 5909.494603] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5914.100982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.114535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5914.124211] Lustre: Skipped 1 previous similar message [ 5917.971233] Lustre: server umount lustre-OST0000 complete [ 5919.205518] LustreError: 195111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.223546] LustreError: 195111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5923.542615] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5923.747536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5927.764890] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5927.918875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5935.570755] LustreError: 195110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5935.588295] LustreError: 195110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5944.805732] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.814479] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5944.827697] Lustre: Skipped 1 previous similar message [ 5944.853717] Lustre: Skipped 3 previous similar messages [ 5948.903570] Lustre: server umount lustre-OST0001 complete [ 5955.045275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.068862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5955.077891] Lustre: Skipped 1 previous similar message [ 5960.073137] Lustre: server umount lustre-MDT0000 complete [ 5964.617931] LustreError: 195133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847652 with bad export cookie 6813393071078362231 [ 5964.623966] LustreError: 195133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5964.626342] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5965.061800] Lustre: server umount lustre-MDT0001 complete [ 5993.812686] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:08:00 (1788847680) [ 6001.253824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.658611] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6001.730599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6002.102385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6007.022950] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6014.918255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.993745] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6015.056591] Lustre: 199567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6020.110159] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6026.703493] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6032.089253] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6038.339653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6038.594627] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6044.319659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6045.723516] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6052.918228] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6058.854074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6058.972413] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6066.977078] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6068.802244] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6075.621597] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6078.954975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.973691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.981918] Lustre: Skipped 2 previous similar messages [ 6084.587938] Lustre: server umount lustre-OST0000 complete [ 6086.626888] LustreError: 201421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.649793] LustreError: 201421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6104.549937] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6104.577020] LustreError: Skipped 1 previous similar message [ 6106.082704] Lustre: server umount lustre-OST0001 complete [ 6112.737992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6112.742136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6112.789210] Lustre: Skipped 2 previous similar messages [ 6112.812889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6112.827857] Lustre: Skipped 6 previous similar messages [ 6117.817591] Lustre: server umount lustre-MDT0000 complete [ 6122.693097] LustreError: 202455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847810 with bad export cookie 6813393071078363638 [ 6122.699624] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6122.708076] LustreError: 202455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6123.272684] Lustre: server umount lustre-MDT0001 complete [ 6151.726463] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:10:38 (1788847838) [ 6168.269275] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 6177.431251] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 6214.139069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.557528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6214.602092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6214.726492] Lustre: lustre-MDT0000: new disk, initializing [ 6214.821379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6214.833330] Lustre: Skipped 3 previous similar messages [ 6214.861969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6219.284602] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6228.168219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.296338] Lustre: 207667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6228.350467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6228.353952] Lustre: Skipped 1 previous similar message [ 6228.434584] Lustre: lustre-MDT0001: new disk, initializing [ 6228.617463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6228.640647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6233.061271] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6241.500737] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6247.249774] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6254.880584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6255.168477] Lustre: lustre-OST0000: new disk, initializing [ 6255.173510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6255.180712] Lustre: 209634:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6260.317431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6260.337099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6260.429924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6260.751165] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6269.351193] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6273.195364] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6273.331902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6277.899622] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6278.123472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6281.192687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6281.200821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6281.222117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6281.222124] Lustre: Skipped 2 previous similar messages [ 6281.251760] Lustre: Skipped 2 previous similar messages [ 6286.532748] Lustre: server umount lustre-OST0000 complete [ 6296.827677] Lustre: server umount lustre-MDT0000 complete [ 6300.917931] LustreError: 207658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847988 with bad export cookie 6813393071078365423 [ 6300.927428] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6300.932514] LustreError: 207658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6301.233607] Lustre: server umount lustre-MDT0001 complete [ 6309.573453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6310.015943] LustreError: 211848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.030401] LustreError: 211848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6314.300488] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6324.054284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.552580] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6336.796192] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6341.772696] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6348.287074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6348.526278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6348.535363] Lustre: Skipped 4 previous similar messages [ 6355.232687] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6362.294634] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6368.003289] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6368.271591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6373.066214] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6373.353211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6380.528333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6380.541324] LustreError: Skipped 1 previous similar message [ 6381.836445] Lustre: server umount lustre-OST0000 complete [ 6387.380713] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6391.826101] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6425.263852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6431.902461] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6439.398610] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6441.087297] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6444.549969] Lustre: server umount lustre-OST0000 complete [ 6450.659839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.695845] Lustre: Skipped 4 previous similar messages [ 6450.712226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6450.734528] Lustre: Skipped 10 previous similar messages [ 6455.058140] Lustre: server umount lustre-MDT0000 complete [ 6455.776317] LustreError: 211848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6455.793922] LustreError: 211848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6458.711035] LustreError: 211829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848146 with bad export cookie 6813393071078366305 [ 6458.718858] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6458.723560] LustreError: 211829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6459.016379] Lustre: server umount lustre-MDT0001 complete [ 6468.177924] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:15:54 (1788848154) [ 6475.689472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6480.392306] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6489.422266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.267666] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6505.213891] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6511.539650] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6518.917702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6525.815852] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6532.836318] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6537.046518] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6537.244738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6541.544972] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6541.734204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6548.324230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.544377] Lustre: lustre-OST0001: new disk, initializing [ 6548.554987] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6548.567574] Lustre: 221180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6555.356893] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6557.320141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6557.338714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6557.450901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6566.066834] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6570.810836] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6571.059828] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6575.849258] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6576.073628] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6578.147587] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6578.174446] LustreError: Skipped 1 previous similar message [ 6584.235555] Lustre: server umount lustre-OST0001 complete [ 6591.033362] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6591.328606] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6595.254359] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6595.437490] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6624.684019] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6624.933215] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6624.943785] Lustre: Skipped 5 previous similar messages [ 6630.387238] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6630.409709] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6630.438991] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6630.446905] Lustre: Skipped 1 previous similar message [ 6631.478910] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6639.087235] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6640.526483] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6645.774925] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6656.991624] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6657.073824] Lustre: server umount lustre-OST0001 complete [ 6687.201138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6687.480365] Lustre: server umount lustre-OST0000 complete [ 6697.394253] Lustre: server umount lustre-MDT0000 complete [ 6700.967528] LustreError: 221179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848388 with bad export cookie 6813393071078367257 [ 6700.981909] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6715.359129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6715.613401] Lustre: server umount lustre-MDT0001 complete [ 6738.548727] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:20:25 (1788848425) [ 6745.911962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.418717] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6746.513431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6752.440038] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6761.569321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.757842] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6761.794451] Lustre: 226430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6766.289308] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6773.395290] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6779.259349] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6785.579474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6785.738990] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6791.495596] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6797.316207] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6806.830676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6806.960521] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6814.406899] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6822.496542] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6823.893779] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6828.578400] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6828.824337] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6833.908378] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6834.145869] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6839.898095] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff971d91b56800.ost_server_uuid 50 [ 6841.382774] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff971d91b56800.ost_server_uuid in IDLE state after 0 sec [ 6842.316346] Lustre: Permanently deactivating lustre-OST0001 [ 6842.335944] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6845.380556] Lustre: Permanently reactivating lustre-OST0001 [ 6852.064886] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6852.072187] LustreError: Skipped 1 previous similar message [ 6852.075810] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.084193] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6852.088559] Lustre: Skipped 10 previous similar messages [ 6852.101993] Lustre: Skipped 22 previous similar messages [ 6854.746916] Lustre: server umount lustre-OST0001 complete [ 6857.208512] LustreError: 228277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.263514] LustreError: 228277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 6865.930150] Lustre: server umount lustre-OST0000 complete [ 6886.367185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6886.808256] Lustre: server umount lustre-MDT0000 complete [ 6890.804020] LustreError: 229302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848578 with bad export cookie 6813393071078368545 [ 6890.815935] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6890.818638] LustreError: 229302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6891.300586] Lustre: server umount lustre-MDT0001 complete [ 6917.215553] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:23:24 (1788848604) [ 6924.464634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.796662] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6924.863950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6924.874734] Lustre: Skipped 5 previous similar messages [ 6930.563590] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6939.931805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.168662] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6940.224435] Lustre: 233124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6945.856595] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6954.035721] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6959.513402] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6965.143435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6965.318303] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6965.338886] Lustre: Permanently deactivating lustre-OST0000 [ 6965.349715] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6971.318789] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6974.557792] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6977.963557] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6985.202705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6985.258246] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6991.539300] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 6994.698325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6999.733087] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7001.886163] Lustre: Permanently reactivating lustre-OST0000 [ 7001.909616] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7001.912774] Lustre: Skipped 2 previous similar messages [ 7010.912807] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7010.915289] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 0000000055ec588e, cur 1788848698 deadline 1788848692 last 1788848662 [ 7010.921525] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7010.945285] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7010.998226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7020.519105] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7020.524622] LustreError: Skipped 4 previous similar messages [ 7030.239736] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7030.372841] Lustre: server umount lustre-OST0001 complete [ 7040.809539] Lustre: server umount lustre-OST0000 complete [ 7051.377513] Lustre: server umount lustre-MDT0000 complete [ 7055.549544] LustreError: 234993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848743 with bad export cookie 6813393071078370197 [ 7055.552124] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7055.561613] LustreError: 234993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7055.872866] Lustre: server umount lustre-MDT0001 complete [ 7068.054800] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 7071.026470] Key type lgssc unregistered [ 7071.361596] LNet: 238005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7071.369915] LNetError: 238005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7071.389261] LNet: Removed LNI 192.168.202.117@tcp [ 7072.337769] Key type .llcrypt unregistered [ 7072.339464] Key type ._llcrypt unregistered [ 7097.477459] Key type ._llcrypt registered [ 7097.481766] Key type .llcrypt registered [ 7097.597801] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7099.236469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7099.360835] alg: No test for adler32 (adler32-zlib) [ 7100.464659] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 7100.789855] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 7102.479333] Key type lgssc registered [ 7103.706784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7113.515787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7113.529973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.997973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7120.099882] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7124.468405] LustreError: 239472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.500733] LustreError: 239472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7129.347649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.474112] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7129.494382] Lustre: 239460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7129.502298] Lustre: Permanently deactivating lustre-MDT0001 [ 7129.516412] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7129.578986] LustreError: 239471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7129.581658] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7129.623939] LustreError: 239502:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7129.642358] LustreError: 239502:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7129.655127] LustreError: 239502:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7129.853569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7134.906772] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7143.181549] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7149.057175] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7155.421351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7155.915160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7159.992406] LustreError: 241314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7160.012362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7162.634578] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7165.410195] LustreError: 241534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7170.530680] LustreError: 241314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7170.678593] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7178.285354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7178.584300] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7183.870502] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7183.883169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7185.749494] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7191.993743] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7196.017678] Lustre: Permanently reactivating lustre-MDT0001 [ 7196.033622] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7196.036534] Lustre: Skipped 2 previous similar messages [ 7202.286943] Lustre: lustre-MDT0001: Client 6c9eef41-027b-4252-98ab-1a4d8ec922c8 (at 192.168.202.17@tcp) reconnecting [ 7207.681895] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7209.087446] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7214.868573] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7215.131458] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7219.559178] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7219.743967] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7221.528333] Lustre: Permanently deactivating lustre-MDT0001 [ 7221.543864] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7221.555279] Lustre: Skipped 2 previous similar messages [ 7228.458569] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7237.092384] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.101451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7237.103548] Lustre: Skipped 2 previous similar messages [ 7237.127035] Lustre: Skipped 3 previous similar messages [ 7237.718996] Lustre: server umount lustre-MDT0000 complete [ 7240.810554] LustreError: 239451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848928 with bad export cookie 5456518727134762263 [ 7240.813231] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7240.827531] LustreError: 239451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7241.154900] Lustre: server umount lustre-MDT0001 complete [ 7244.629478] Lustre: server umount lustre-OST0000 complete [ 7254.447296] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:29:01 (1788848941) [ 7265.004441] Lustre: server umount lustre-OST0001 complete [ 7271.516679] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 7281.641500] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7312.912454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7313.074482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7313.080579] Lustre: Skipped 2 previous similar messages [ 7313.103083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7313.188778] Lustre: lustre-MDT0000: new disk, initializing [ 7313.261989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.285273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7317.187863] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7326.161849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7326.260607] Lustre: 247742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7326.304415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7326.306491] Lustre: Skipped 1 previous similar message [ 7326.358839] Lustre: lustre-MDT0001: new disk, initializing [ 7326.445539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7326.482375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7326.495385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7329.887624] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7338.458637] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7344.493692] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7350.976133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7351.233887] Lustre: lustre-OST0000: new disk, initializing [ 7351.240482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7351.249413] Lustre: 249708:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7351.334450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7356.673061] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7358.086851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7358.115246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7358.224756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7364.813262] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7366.318816] LustreError: 247747:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7374.319816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.481955] Lustre: lustre-OST0001: new disk, initializing [ 7374.489805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7374.495645] Lustre: 250822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7376.351269] LustreError: 247747:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7376.368803] LustreError: 250379:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7379.971136] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7380.099603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7380.117672] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7380.186975] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7386.409639] LustreError: 250379:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7388.009782] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7389.363499] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7393.260181] Lustre: lustre-OST0001: Not available for connect from 192.168.202.17@tcp (stopping) [ 7395.811162] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7395.816813] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7395.822921] Lustre: Skipped 1 previous similar message [ 7395.830105] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7397.221345] Lustre: server umount lustre-OST0001 complete [ 7398.360579] LustreError: 249692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.380750] LustreError: 249692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7414.754900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7414.757630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7414.761878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7414.762014] Lustre: Skipped 1 previous similar message [ 7414.772436] Lustre: Skipped 2 previous similar messages [ 7416.295070] LustreError: 249689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7416.328708] LustreError: 249689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7418.817250] Lustre: server umount lustre-OST0000 complete [ 7423.971969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7423.988705] LustreError: Skipped 1 previous similar message [ 7423.994781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7424.020401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7424.023763] Lustre: Skipped 2 previous similar messages [ 7436.258344] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7436.438579] Lustre: server umount lustre-MDT0000 complete [ 7439.714074] LustreError: 247734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849127 with bad export cookie 5456518727134764860 [ 7439.723941] LustreError: 247734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7439.727910] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7454.175482] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7454.527483] Lustre: server umount lustre-MDT0001 complete [ 7466.018459] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 7468.641514] Key type lgssc unregistered [ 7468.983498] LNet: 252859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7468.988023] LNetError: 252859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7469.007322] LNet: Removed LNI 192.168.202.117@tcp [ 7469.918557] Key type .llcrypt unregistered [ 7469.920647] Key type ._llcrypt unregistered [ 7507.464124] Key type ._llcrypt registered [ 7507.466229] Key type .llcrypt registered [ 7507.625860] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7508.932560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7509.055594] alg: No test for adler32 (adler32-zlib) [ 7510.048064] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 7510.262480] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 7511.911196] Key type lgssc registered [ 7513.025734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7520.556444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7520.594405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.850940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7521.883724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7522.031089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.400224] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7534.628844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7534.747782] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7534.799955] Lustre: 255061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7535.080590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7539.703915] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7547.094493] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7551.841452] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7557.468616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7557.712864] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7557.903037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.461218] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7566.938401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7571.497107] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7623.136239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7623.141757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7623.144917] LustreError: Skipped 1 previous similar message [ 7623.159311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7623.165811] Lustre: Skipped 1 previous similar message [ 7626.702448] Lustre: server umount lustre-OST0000 complete [ 7631.419692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7643.963209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7644.248808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7649.759295] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7656.574652] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7663.656231] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7663.668654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7663.675785] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7663.685376] Lustre: Skipped 1 previous similar message [ 7694.304623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7694.304996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7694.312403] LustreError: Skipped 1 previous similar message [ 7694.314129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7694.334985] Lustre: Skipped 2 previous similar messages [ 7699.342589] Lustre: server umount lustre-OST0000 complete [ 7704.032750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7704.037377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7704.051714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7704.055157] Lustre: Skipped 1 previous similar message [ 7708.594403] Lustre: server umount lustre-MDT0000 complete [ 7711.589728] LustreError: 255053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849399 with bad export cookie 7976851803809143936 [ 7711.590981] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7711.607035] LustreError: 255053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7711.917192] Lustre: server umount lustre-MDT0001 complete [ 7721.143683] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 7723.620211] Key type lgssc unregistered [ 7723.865670] LNet: 260486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7723.871674] LNetError: 260486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7723.885529] LNet: Removed LNI 192.168.202.117@tcp [ 7724.713941] Key type .llcrypt unregistered [ 7724.717174] Key type ._llcrypt unregistered [ 7747.004487] Key type ._llcrypt registered [ 7747.005660] Key type .llcrypt registered [ 7747.208653] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7748.714495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7748.740616] alg: No test for adler32 (adler32-zlib) [ 7749.888044] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 7750.163225] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 7751.879467] Key type lgssc registered [ 7753.230379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7761.735800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7761.763305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7763.287107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.932877] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7773.666183] LustreError: 261892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7773.689252] LustreError: 261892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7775.622731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7775.959067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7780.556493] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7787.201711] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7792.153284] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7798.556401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7798.896196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7805.258650] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7809.144940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7812.187210] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7835.107677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.109479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7835.127184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7838.522325] Lustre: server umount lustre-OST0000 complete [ 7842.785512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7842.801475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7842.815542] Lustre: Skipped 1 previous similar message [ 7842.821089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7842.826419] Lustre: Skipped 1 previous similar message [ 7845.346405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7845.347562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7847.799909] Lustre: server umount lustre-MDT0000 complete [ 7850.469327] LustreError: 263724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7850.482360] LustreError: 263724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7850.524373] LustreError: 261874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849538 with bad export cookie 11073259149107462246 [ 7850.532894] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7850.536578] LustreError: 261874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7850.778968] Lustre: server umount lustre-MDT0001 complete [ 7858.943542] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 7861.033484] Key type lgssc unregistered [ 7861.237623] LNet: 266194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7861.243834] LNetError: 266194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7861.258871] LNet: Removed LNI 192.168.202.117@tcp [ 7861.893644] Key type .llcrypt unregistered [ 7861.895217] Key type ._llcrypt unregistered [ 7875.051899] Key type ._llcrypt registered [ 7875.055319] Key type .llcrypt registered [ 7875.143655] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7875.710698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7875.728739] alg: No test for adler32 (adler32-zlib) [ 7876.685587] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 7876.856673] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 7878.512373] Key type lgssc registered [ 7879.293710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7885.239262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7885.250844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.619329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7890.474250] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7897.062050] LustreError: 267391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.076113] LustreError: 267391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7897.627036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.846059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7901.657856] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7908.083200] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7913.152888] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7918.965967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7919.304055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7924.678449] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 7931.520862] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7931.592536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7942.117703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7942.120258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7942.127688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7942.131311] Lustre: Skipped 1 previous similar message [ 7947.235490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7947.250407] Lustre: Skipped 2 previous similar messages [ 7947.924832] Lustre: server umount lustre-OST0000 complete [ 7952.352703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.361983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7957.473384] LustreError: 267387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7957.484096] LustreError: 267387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7957.545386] Lustre: server umount lustre-MDT0000 complete [ 7960.546664] LustreError: 267372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849648 with bad export cookie 12850528094168247000 [ 7960.548151] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.558168] LustreError: 267372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7960.867789] Lustre: server umount lustre-MDT0001 complete [ 7969.828529] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 7972.053978] Key type lgssc unregistered [ 7972.261826] LNet: 271064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7972.265744] LNetError: 271064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7972.278933] LNet: Removed LNI 192.168.202.117@tcp [ 7972.993457] Key type .llcrypt unregistered [ 7972.996479] Key type ._llcrypt unregistered [ 7992.119168] Key type ._llcrypt registered [ 7992.121330] Key type .llcrypt registered [ 7992.222285] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 7993.154957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7993.222441] alg: No test for adler32 (adler32-zlib) [ 7994.234707] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 7994.490499] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 7996.152218] Key type lgssc registered [ 7997.227223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8003.705665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8003.723187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8005.028139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.262594] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8014.318056] LustreError: 272455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.329495] LustreError: 272455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8015.279756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8015.539986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8019.044619] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8025.181421] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8029.973664] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8036.348145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8036.675571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8043.363894] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8050.588609] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8051.002587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8071.651770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8071.664038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8071.672026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8071.683041] Lustre: Skipped 1 previous similar message [ 8074.671460] Lustre: server umount lustre-OST0000 complete [ 8081.888721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.899312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8081.901538] Lustre: Skipped 2 previous similar messages [ 8084.012507] Lustre: server umount lustre-MDT0000 complete [ 8087.008777] LustreError: 272451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8087.028185] LustreError: 272451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8087.085241] LustreError: 276007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849774 with bad export cookie 9749277618768134182 [ 8087.091032] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8087.094651] LustreError: 276007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8087.487519] Lustre: server umount lustre-MDT0001 complete [ 8096.750554] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 8099.025084] Key type lgssc unregistered [ 8099.230015] LNet: 276758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8099.236784] LNetError: 276758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8099.255127] LNet: Removed LNI 192.168.202.117@tcp [ 8099.925531] Key type .llcrypt unregistered [ 8099.929321] Key type ._llcrypt unregistered [ 8114.339360] Key type ._llcrypt registered [ 8114.342470] Key type .llcrypt registered [ 8114.476355] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8115.386854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8115.558837] alg: No test for adler32 (adler32-zlib) [ 8116.687854] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 8116.963221] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 8118.591527] Key type lgssc registered [ 8119.544232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8125.646588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8125.662916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.027318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.763854] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8137.189817] LustreError: 277970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8137.206338] LustreError: 277970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8137.481858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.691955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8141.119668] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8146.614977] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8151.142609] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8156.903912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8157.257471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.814678] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8169.769138] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8172.598528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8187.880199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.881163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8187.887684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8187.895700] Lustre: Skipped 1 previous similar message [ 8187.917085] LustreError: Skipped 1 previous similar message [ 8190.180553] Lustre: server umount lustre-OST0000 complete [ 8194.016833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8194.023266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.035885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8194.039778] Lustre: Skipped 1 previous similar message [ 8198.113460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8198.116311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8198.146065] Lustre: Skipped 1 previous similar message [ 8203.233731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8203.237553] Lustre: Skipped 1 previous similar message [ 8207.332839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8207.338759] Lustre: Skipped 1 previous similar message [ 8208.351146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8208.623516] Lustre: server umount lustre-MDT0000 complete [ 8212.113115] LustreError: 277935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849899 with bad export cookie 1423090609622063206 [ 8212.114652] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8212.125378] LustreError: 277935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8212.405690] Lustre: server umount lustre-MDT0001 complete [ 8220.769149] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 8223.204391] Key type lgssc unregistered [ 8223.465654] LNet: 281668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8223.470155] LNetError: 281668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8223.494686] LNet: Removed LNI 192.168.202.117@tcp [ 8224.216149] Key type .llcrypt unregistered [ 8224.218384] Key type ._llcrypt unregistered [ 8233.543247] Key type ._llcrypt registered [ 8233.545193] Key type .llcrypt registered [ 8233.666939] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8247.284528] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 8259.212422] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8259.992424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8260.029715] alg: No test for adler32 (adler32-zlib) [ 8261.005814] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 8261.196529] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 8262.864861] Key type lgssc registered [ 8264.040697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8292.341299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8292.364375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.616876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8293.645832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8293.708666] Lustre: lustre-MDT0000: new disk, initializing [ 8293.761330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8293.771696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8298.001335] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8307.140930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8307.285945] Lustre: 285193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8307.368725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8307.377582] Lustre: Skipped 1 previous similar message [ 8307.458217] Lustre: lustre-MDT0001: new disk, initializing [ 8307.535116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8307.565562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8307.578386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8311.608619] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8319.887928] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8325.839522] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8332.374091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8332.645698] Lustre: lustre-OST0000: new disk, initializing [ 8332.650847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8332.657393] Lustre: 287159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8332.742473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.998455] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8339.012916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8339.029777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8339.137563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8346.745091] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8350.476352] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8350.657640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8354.827126] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8355.084622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8359.904244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8359.906532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8359.912227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8359.916072] Lustre: Skipped 1 previous similar message [ 8362.793155] Lustre: server umount lustre-OST0000 complete [ 8369.132115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8369.139016] LustreError: Skipped 1 previous similar message [ 8369.142861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8369.154664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8369.157597] Lustre: Skipped 1 previous similar message [ 8370.147103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.162148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.167336] Lustre: Skipped 1 previous similar message [ 8375.265785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.269701] Lustre: Skipped 1 previous similar message [ 8379.390278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.383233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8380.588325] Lustre: server umount lustre-MDT0000 complete [ 8383.531042] LustreError: 285185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850071 with bad export cookie 13064531508661311935 [ 8383.544557] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8383.817922] Lustre: server umount lustre-MDT0001 complete [ 8390.856961] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:47:57 (1788850077) [ 8396.110701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8396.525978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8400.605684] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8401.897033] LustreError: 289573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8401.923609] LustreError: 289573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8407.009469] LustreError: 289574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8407.530789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.764471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8411.253398] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8416.690335] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8420.930636] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8425.662543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8425.894810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8430.113397] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8435.064725] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8453.615574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8453.617145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8453.621152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8453.621162] Lustre: Skipped 1 previous similar message [ 8453.623206] LustreError: Skipped 1 previous similar message [ 8453.674377] Lustre: Skipped 1 previous similar message [ 8459.636360] Lustre: server umount lustre-OST0000 complete [ 8463.846232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8469.019973] Lustre: server umount lustre-MDT0000 complete [ 8472.061801] LustreError: 292174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850159 with bad export cookie 13064531508661312817 [ 8472.063149] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8472.070167] LustreError: 292174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8472.353312] Lustre: server umount lustre-MDT0001 complete [ 8481.068583] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 8483.405438] Key type lgssc unregistered [ 8483.656724] LNet: 293126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8483.661477] LNetError: 293126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8483.681968] LNet: Removed LNI 192.168.202.117@tcp [ 8484.305337] Key type .llcrypt unregistered [ 8484.307126] Key type ._llcrypt unregistered [ 8506.922510] Key type ._llcrypt registered [ 8506.924900] Key type .llcrypt registered [ 8507.007664] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 8520.904471] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8522.034200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8522.056363] alg: No test for adler32 (adler32-zlib) [ 8523.086184] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 8523.312984] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 8524.991188] Key type lgssc registered [ 8525.927189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8557.265830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8557.285203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8558.520830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8558.549195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8558.624664] Lustre: lustre-MDT0000: new disk, initializing [ 8558.699463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8558.711230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8561.743243] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8569.709391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.816590] Lustre: 296948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8569.864707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8569.868985] Lustre: Skipped 1 previous similar message [ 8569.938505] Lustre: lustre-MDT0001: new disk, initializing [ 8570.031877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8570.066135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8570.076242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8573.270922] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8580.537546] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8585.350700] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8590.740827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8590.962804] Lustre: lustre-OST2710: new disk, initializing [ 8590.965890] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8590.971531] Lustre: 298911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8591.061281] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8595.873530] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8597.064770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8597.078422] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8597.167089] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8603.264481] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8608.741879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8608.869781] Lustre: lustre-OST03e8: new disk, initializing [ 8608.872775] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8608.880951] Lustre: 299977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8608.954644] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8614.111399] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8616.029210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8616.046069] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8616.106324] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8619.933847] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8624.802718] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8624.958646] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8628.318570] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8628.478840] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8632.251303] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8632.408256] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8635.498453] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8635.644553] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8642.016978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8642.020537] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.024118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.038447] Lustre: Skipped 2 previous similar messages [ 8645.210298] Lustre: server umount lustre-MDT0000 complete [ 8647.926623] LustreError: 296939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850335 with bad export cookie 16634727656460053485 [ 8647.934129] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.173371] Lustre: server umount lustre-MDT0001 complete [ 8661.563339] Lustre: server umount lustre-OST2710 complete [ 8674.668345] Lustre: server umount lustre-OST03e8 complete [ 8687.216243] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 8694.535336] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8719.675182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.816990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8719.833121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8719.885763] Lustre: lustre-MDT0000: new disk, initializing [ 8719.931419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.942572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8723.162532] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8730.825162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8730.902518] Lustre: 305223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8730.940361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8730.945468] Lustre: Skipped 1 previous similar message [ 8731.021662] Lustre: lustre-MDT0001: new disk, initializing [ 8731.105444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8731.139099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8731.152833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8733.956398] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8740.243982] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8744.455414] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8749.002300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8749.234629] Lustre: lustre-OST0000: new disk, initializing [ 8749.238468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8749.244458] Lustre: 307187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8749.331284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8753.563216] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8756.774703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8756.784700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8756.878445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8760.193524] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8763.041463] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8763.189317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8765.960190] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8766.076080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8767.456484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8767.459697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.467164] Lustre: Skipped 1 previous similar message [ 8767.474444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8767.483171] Lustre: Skipped 3 previous similar messages [ 8772.066758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8772.070043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8772.075944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.369543] Lustre: server umount lustre-OST0000 complete [ 8777.187485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8777.195512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.199965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8777.208457] Lustre: Skipped 1 previous similar message [ 8777.214177] Lustre: Skipped 2 previous similar messages [ 8781.635950] LustreError: 308726:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (000000002dbcedba) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 8781.686536] Lustre: server umount lustre-MDT0000 complete [ 8782.309632] LustreError: 305234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.326762] LustreError: 305234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8784.408171] LustreError: 305214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850472 with bad export cookie 16634727656460055158 [ 8784.411797] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8784.421889] LustreError: 305214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8784.621794] Lustre: server umount lustre-MDT0001 complete [ 8790.233484] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:54:37 (1788850477) [ 8791.186188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8792.393173] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:54:39 (1788850479) [ 8794.447419] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8812.763272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.906406] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8812.923461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8813.062960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8816.046762] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8821.870882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.934955] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8821.970464] Lustre: 311041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8825.515344] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8844.274643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8844.414497] Lustre: Denying initial registration attempt from nid 192.168.202.117@tcp, specified as failover [ 8844.421502] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8844.428594] LustreError: 312494:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 8844.441558] LustreError: 312494:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8844.447462] LustreError: 312494:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8844.484833] Lustre: server umount lustre-OST0000 complete [ 8844.486602] LustreError: 312494:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8850.402898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8850.416944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8850.421731] Lustre: Skipped 1 previous similar message [ 8855.175363] Lustre: server umount lustre-MDT0000 complete [ 8855.520587] LustreError: 311052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.536935] LustreError: 311052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8857.699987] LustreError: 311032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850545 with bad export cookie 16634727656460056047 [ 8857.701774] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8857.706216] LustreError: 311032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8857.908988] Lustre: server umount lustre-MDT0001 complete [ 8865.056974] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 8867.105842] Key type lgssc unregistered [ 8867.346027] LNet: 313612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8867.350809] LNetError: 313612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8867.362717] LNet: Removed LNI 192.168.202.117@tcp [ 8867.933178] Key type .llcrypt unregistered [ 8867.934593] Key type ._llcrypt unregistered [ 8874.627339] Key type ._llcrypt registered [ 8874.628937] Key type .llcrypt registered [ 8874.693712] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8875.272416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8875.294905] alg: No test for adler32 (adler32-zlib) [ 8876.199097] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 8876.340196] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 8877.943137] Key type lgssc registered [ 8878.558038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8911.675868] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 8917.661900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8917.683989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.893496] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8918.906453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8919.029735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8922.317773] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8928.497079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8928.621652] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8928.643489] Lustre: 316348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8928.813370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8932.449887] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8938.279903] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8942.462626] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8960.744727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8960.926769] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8961.127458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8965.543242] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 8971.954404] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8976.363676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8976.377989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8976.381910] Lustre: Skipped 1 previous similar message [ 8976.389902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8980.927937] Lustre: server umount lustre-OST0000 complete [ 8986.594463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8986.600703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.619904] Lustre: Skipped 1 previous similar message [ 8986.638619] Lustre: Skipped 2 previous similar messages [ 8991.721056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8991.732193] Lustre: Skipped 1 previous similar message [ 8992.237511] Lustre: server umount lustre-MDT0000 complete [ 8997.441072] LustreError: 317088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850685 with bad export cookie 6975805049179018011 [ 8997.447221] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8997.457512] LustreError: 317088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8998.319467] Lustre: server umount lustre-MDT0001 complete [ 9015.347480] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9020.720963] Key type lgssc unregistered [ 9021.153813] LNet: 319919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9021.161251] LNetError: 319919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9021.193245] LNet: Removed LNI 192.168.202.117@tcp [ 9022.628872] Key type .llcrypt unregistered [ 9022.630906] Key type ._llcrypt unregistered [ 9052.137790] Key type ._llcrypt registered [ 9052.146253] Key type .llcrypt registered [ 9052.358557] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 9053.810452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9053.870249] alg: No test for adler32 (adler32-zlib) [ 9054.997917] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 9055.289621] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 9057.039167] Key type lgssc registered [ 9058.009945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9065.214901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9065.241124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.691403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9071.415590] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9076.193590] LustreError: 321334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9076.211172] LustreError: 321334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9078.667350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.923862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9083.025523] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9089.758041] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9095.005138] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9103.029727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9103.443942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9109.501095] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9116.583458] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9124.834053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9124.843262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9124.861752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9127.395980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9127.397769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9128.426844] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.17@tcp (stopping) [ 9128.437331] Lustre: Skipped 2 previous similar messages [ 9129.942356] Lustre: server umount lustre-MDT0000 complete [ 9132.529499] LustreError: 321334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.550701] LustreError: 321334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9133.536692] LustreError: 323149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9133.674043] LustreError: 321315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850821 with bad export cookie 11877192255016158178 [ 9133.680538] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9133.690632] LustreError: 321315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9137.634142] LustreError: 321331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9137.650722] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9137.666344] Lustre: Skipped 1 previous similar message [ 9137.671159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9143.761841] LustreError: 323545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9143.771986] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.17@tcp (stopping) [ 9143.774155] LustreError: 323545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9143.797105] Lustre: Skipped 3 previous similar messages [ 9147.871520] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9148.302811] Lustre: server umount lustre-MDT0001 complete [ 9154.358175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9162.530032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.592453] LustreError: 320644:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d3d2e3b4 x1875745956736640/t0(0) o250->MGC192.168.202.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9178.903188] LustreError: 323166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9178.928281] LustreError: 323166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9178.994871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.057413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9183.650393] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9191.434622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.801570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9195.989471] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9199.087931] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9203.078734] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9205.038275] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9213.672562] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9215.976595] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9222.635308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9222.643430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.670079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9227.034719] Lustre: server umount lustre-OST0000 complete [ 9232.865078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9232.869700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.879418] Lustre: Skipped 1 previous similar message [ 9238.315758] Lustre: server umount lustre-MDT0000 complete [ 9240.035546] LustreError: 324945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.060119] LustreError: 324945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9242.195105] LustreError: 321315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850930 with bad export cookie 11877192255016166018 [ 9242.199293] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9242.204616] LustreError: 321315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9242.539251] Lustre: server umount lustre-MDT0001 complete [ 9254.516914] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9257.828657] Key type lgssc unregistered [ 9258.181955] LNet: 327613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9258.188141] LNetError: 327613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9258.202324] LNet: Removed LNI 192.168.202.117@tcp [ 9259.055727] Key type .llcrypt unregistered [ 9259.057796] Key type ._llcrypt unregistered [ 9284.949876] Key type ._llcrypt registered [ 9284.951227] Key type .llcrypt registered [ 9285.032500] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 9286.222896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9286.239147] alg: No test for adler32 (adler32-zlib) [ 9287.358472] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 9287.621371] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 9289.295320] Key type lgssc registered [ 9290.624623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9298.518777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9298.534355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.927029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9304.438437] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9310.178342] LustreError: 329032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9310.189051] LustreError: 329032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9312.462946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9312.736443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9316.609578] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9323.825899] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9329.630443] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9333.223287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9333.237140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9333.255813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.289140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.293517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9337.378749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9337.385665] Lustre: Skipped 2 previous similar messages [ 9339.467985] Lustre: server umount lustre-MDT0000 complete [ 9342.449429] LustreError: 329033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9342.485071] LustreError: 329033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9343.425646] LustreError: 329014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851031 with bad export cookie 16854632640835188293 [ 9343.429178] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9343.440074] LustreError: 329014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9343.971888] Lustre: server umount lustre-MDT0001 complete [ 9360.023365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.429368] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9360.544119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9360.871199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9366.564964] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9375.540115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.611358] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9375.664985] Lustre: 331620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9376.031950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9381.515425] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9389.107789] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9395.128870] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9404.655766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9404.923394] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9404.935188] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9404.941796] LustreError: 333510:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9404.948342] LustreError: 333510:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9404.952437] LustreError: 333510:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9405.028924] Lustre: server umount lustre-OST0000 complete [ 9405.034275] LustreError: 333510:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9415.531379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9415.805773] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9416.156715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.711607] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9425.008811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9430.713757] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9438.595383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9438.691947] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9438.821169] Lustre: lustre-OST0001: new disk, initializing [ 9438.833136] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9438.840192] Lustre: 335052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9438.937712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9445.720151] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9446.117735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9446.168423] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9446.330777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9455.351632] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9461.731042] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9461.754592] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.764586] LustreError: Skipped 1 previous similar message [ 9461.765858] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9461.824495] Lustre: Skipped 1 previous similar message [ 9464.600783] Lustre: server umount lustre-OST0001 complete [ 9466.854901] LustreError: 333977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9466.875661] LustreError: 333977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9470.945470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9470.952184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9470.969805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9470.980656] Lustre: Skipped 1 previous similar message [ 9472.011446] LustreError: 335344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9472.047295] LustreError: 335344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9475.296554] Lustre: server umount lustre-OST0000 complete [ 9482.220420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9482.251024] Lustre: Skipped 1 previous similar message [ 9482.257086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9482.266312] Lustre: Skipped 3 previous similar messages [ 9485.948583] Lustre: server umount lustre-MDT0000 complete [ 9487.329669] LustreError: 332614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9487.358882] LustreError: 332614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9490.433126] LustreError: 333992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851178 with bad export cookie 16854632640835188846 [ 9490.447857] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9490.450263] LustreError: 333992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9491.142656] Lustre: server umount lustre-MDT0001 complete [ 9503.968552] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [ 9506.615886] Key type lgssc unregistered [ 9507.070769] LNet: 337011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9507.089156] LNetError: 337011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9507.129622] LNet: Removed LNI 192.168.202.117@tcp [ 9507.800207] Key type .llcrypt unregistered [ 9507.801312] Key type ._llcrypt unregistered [ 9579.044710] Key type ._llcrypt registered [ 9579.047224] Key type .llcrypt registered [ 9579.179153] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [ 9597.414420] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [ 9599.510192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9599.537280] alg: No test for adler32 (adler32-zlib) [ 9600.802861] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [ 9600.996774] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [ 9602.647207] Key type lgssc registered [ 9603.945215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9640.101332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9640.123686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.545529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9641.611409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9641.736174] Lustre: lustre-MDT0000: new disk, initializing [ 9641.823876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9641.842940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9646.670193] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9656.659891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.750349] Lustre: 341943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9656.792490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9656.795795] Lustre: Skipped 1 previous similar message [ 9656.856786] Lustre: lustre-MDT0001: new disk, initializing [ 9657.038175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9657.080371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9657.103892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9661.351542] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9670.071934] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9676.103713] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9683.358702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9683.605884] Lustre: lustre-OST0000: new disk, initializing [ 9683.607714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9683.610798] Lustre: 343913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9683.654288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.821727] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9693.254106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9693.277264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9693.435020] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9700.017369] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9706.861833] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9707.254032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9712.834562] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9713.181973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9719.273678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9719.279194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9719.280927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9719.301940] LustreError: Skipped 1 previous similar message [ 9719.318364] Lustre: Skipped 1 previous similar message [ 9721.152832] Lustre: server umount lustre-OST0000 complete [ 9728.994292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9729.011390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9729.028771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9729.033066] Lustre: Skipped 1 previous similar message [ 9731.662862] Lustre: server umount lustre-MDT0000 complete [ 9735.796731] LustreError: 341935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851423 with bad export cookie 5098700492454227389 [ 9735.799224] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9735.819976] LustreError: 341935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9736.124844] Lustre: server umount lustre-MDT0001 complete [ 9743.565235] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:10:30 (1788851430) [ 9751.720989] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:10:38 (1788851438) [ 9762.603090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9763.131532] LustreError: 346669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.144109] LustreError: 346669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9763.180796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9768.428806] LustreError: 346670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9768.541241] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9772.519967] LustreError: 346669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9777.635115] LustreError: 346670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9778.275597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9778.739364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9783.709411] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9789.691927] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9796.643817] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9804.514459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9804.858603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9813.577953] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9820.624505] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9832.492887] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9834.878868] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9837.026884] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9838.567579] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9840.638837] Lustre: Failing over lustre-MDT0000 [ 9840.858512] Lustre: server umount lustre-MDT0000 complete [ 9843.203268] LustreError: 346666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9844.706383] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9844.731364] Lustre: Skipped 3 previous similar messages [ 9853.400340] LustreError: 346664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.441322] LustreError: 346664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9860.740175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9860.883332] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9861.064290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.105625] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9864.660259] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9865.775244] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9866.217484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9866.239858] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9866.291569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9876.938354] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9878.763132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9883.118594] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9885.557895] Lustre: Failing over lustre-MDT0000 [ 9885.958556] Lustre: server umount lustre-MDT0000 complete [ 9886.691671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.697831] LustreError: 346666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.706836] Lustre: Skipped 1 previous similar message [ 9886.708048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9886.748247] LustreError: 346666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 9903.071330] Lustre: 340164:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851574/real 1788851574] req@000000007aa52fc8 x1875746528258816/t0(0) o400->MGC192.168.202.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788851590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9903.101968] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9907.206932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.723445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9912.804366] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9916.374144] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9917.973407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9917.985415] Lustre: Skipped 2 previous similar messages [ 9918.036431] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9918.096131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9918.539701] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [ 9928.730266] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9931.084217] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9938.401923] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9938.402740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9938.440122] Lustre: Skipped 2 previous similar messages [ 9938.458512] Lustre: Skipped 4 previous similar messages [ 9943.522535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9943.527483] Lustre: Skipped 2 previous similar messages [ 9944.107337] Lustre: server umount lustre-MDT0000 complete [ 9948.640427] LustreError: 346669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9948.684360] LustreError: 346669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 9949.858786] LustreError: 346651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851637 with bad export cookie 5098700492454230091 [ 9949.868278] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9949.870584] LustreError: 346651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9950.447419] Lustre: server umount lustre-MDT0001 complete [ 9966.344240] Lustre: server umount lustre-OST0000 complete [ 9984.630414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9985.384898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9991.878644] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10000.209750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10005.298590] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10012.259606] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10018.791838] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10025.901087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10031.612560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10034.373904] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10042.324535] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10056.166476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10056.169116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10056.180808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10056.183854] Lustre: Skipped 2 previous similar messages [10059.236372] Lustre: server umount lustre-OST0000 complete [10066.401399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10066.415420] Lustre: Skipped 2 previous similar messages [10069.894565] Lustre: server umount lustre-MDT0000 complete [10071.521867] LustreError: 353030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10071.549921] LustreError: 353030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10074.267359] LustreError: 353011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851762 with bad export cookie 5098700492454230840 [10074.281325] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10074.669569] Lustre: server umount lustre-MDT0001 complete [10087.792308] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [10091.093955] Key type lgssc unregistered [10091.371604] LNet: 356560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10091.381462] LNetError: 356560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10091.418228] LNet: Removed LNI 192.168.202.117@tcp [10092.385953] Key type .llcrypt unregistered [10092.392461] Key type ._llcrypt unregistered [10119.608768] Key type ._llcrypt registered [10119.612175] Key type .llcrypt registered [10119.726399] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [10120.879612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10120.920696] alg: No test for adler32 (adler32-zlib) [10122.288425] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [10122.584965] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [10124.231179] Key type lgssc registered [10125.906045] Lustre: Echo OBD driver; http://www.lustre.org/ [10135.240729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10135.280502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10136.808683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10142.219663] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10146.289694] LustreError: 358077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10146.311756] LustreError: 358077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10151.115912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10151.396950] LustreError: 358076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10151.788193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10157.652984] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10166.494572] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10173.789416] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10180.130598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10180.777328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10181.840070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10189.717479] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10199.462507] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10211.841870] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10211.854834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10211.874066] Lustre: Skipped 2 previous similar messages [10211.896025] Lustre: Skipped 2 previous similar messages [10212.188820] Lustre: server umount lustre-MDT0000 complete [10215.916854] LustreError: 358056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851903 with bad export cookie 15300504137479453302 [10215.925682] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10215.945357] LustreError: 358056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10216.378039] Lustre: server umount lustre-MDT0001 complete [10232.408710] Lustre: server umount lustre-OST0000 complete [10245.047402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.416869] LustreError: 361788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10245.510252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10249.634585] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10250.724607] LustreError: 361789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.874476] LustreError: 361788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10259.984526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10260.368921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10266.531458] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10274.852597] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10281.257454] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10289.695626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10290.127429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10291.131095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10297.884800] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10306.463789] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10320.866392] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.874162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10320.901212] Lustre: Skipped 1 previous similar message [10320.918827] Lustre: Skipped 2 previous similar messages [10325.985483] LustreError: 362517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10326.006578] LustreError: 362517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10327.069135] Lustre: server umount lustre-MDT0000 complete [10330.522618] LustreError: 361768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852018 with bad export cookie 15300504137479454415 [10330.526791] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10330.529538] LustreError: 361768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10330.692767] Lustre: server umount lustre-MDT0001 complete [10343.720762] Lustre: server umount lustre-OST0000 complete [10362.755682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10363.264072] LustreError: 365701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10363.321503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10367.938191] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10376.738037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.106764] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10388.351711] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10395.181372] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10403.703072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10404.382361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10404.399294] Lustre: Skipped 1 previous similar message [10408.517243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10412.287929] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10424.522718] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10432.488116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10432.485368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10432.513504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10432.555753] Lustre: Skipped 2 previous similar messages [10432.599223] Lustre: Skipped 1 previous similar message [10437.603023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10437.609584] Lustre: Skipped 1 previous similar message [10438.197679] Lustre: server umount lustre-OST0000 complete [10443.745728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10443.764543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10443.787889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10447.848152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10449.539201] Lustre: server umount lustre-MDT0000 complete [10452.961074] LustreError: 366429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10452.991740] LustreError: 366429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10453.795026] LustreError: 365682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852141 with bad export cookie 15300504137479455437 [10453.815579] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10454.114834] Lustre: server umount lustre-MDT0001 complete [10471.555170] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [10475.774481] Key type lgssc unregistered [10476.075371] LNet: 369236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10476.084492] LNetError: 369236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10476.101433] LNet: Removed LNI 192.168.202.117@tcp [10477.245717] Key type .llcrypt unregistered [10477.248732] Key type ._llcrypt unregistered [10506.191090] Key type ._llcrypt registered [10506.197686] Key type .llcrypt registered [10506.334594] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [10507.552544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10507.611787] alg: No test for adler32 (adler32-zlib) [10508.727979] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [10509.122493] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [10510.807179] Key type lgssc registered [10512.067282] Lustre: Echo OBD driver; http://www.lustre.org/ [10520.346730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10520.372612] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10520.383516] LustreError: 370675:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10520.424144] LustreError: 370675:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10520.433408] LustreError: 370675:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10520.437469] LustreError: 370675:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10520.447702] LustreError: 370675:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10528.878568] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10528.886047] LustreError: 371072:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10528.901845] LustreError: 371072:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10528.906637] LustreError: 371072:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10528.915960] LustreError: 371072:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10528.925618] LustreError: 371072:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10548.067401] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [10550.517910] Key type lgssc unregistered [10550.848558] LNet: 372074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10550.857820] LNetError: 372074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10550.874808] LNet: Removed LNI 192.168.202.117@tcp [10551.918312] Key type .llcrypt unregistered [10551.920762] Key type ._llcrypt unregistered [10572.040735] Key type ._llcrypt registered [10572.046892] Key type .llcrypt registered [10572.188289] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [10589.101840] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [10591.054202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10591.080869] alg: No test for adler32 (adler32-zlib) [10592.280738] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [10592.616435] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [10594.403506] Key type lgssc registered [10595.987705] Lustre: Echo OBD driver; http://www.lustre.org/ [10635.065925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10635.104789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10636.403116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10636.454481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10636.542471] Lustre: lustre-MDT0000: new disk, initializing [10636.669710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10636.696928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10641.025599] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10651.311631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10651.433775] Lustre: 375227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10651.496397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10651.499994] Lustre: Skipped 1 previous similar message [10651.562618] Lustre: lustre-MDT0001: new disk, initializing [10651.640123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10651.672993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10651.688965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10655.906667] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10665.048857] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10670.645069] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10677.295590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10677.489466] Lustre: lustre-OST0000: new disk, initializing [10677.490956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10677.493211] Lustre: 377196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10677.528872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10684.023778] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10686.518185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10686.533848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10686.620259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10693.625887] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10699.683624] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10699.943938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10705.986184] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10706.299799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10712.564228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10712.567233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10712.569380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10712.578403] Lustre: Skipped 1 previous similar message [10715.353383] Lustre: server umount lustre-OST0000 complete [10722.793150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10722.820237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10722.826806] Lustre: Skipped 2 previous similar messages [10726.116629] Lustre: server umount lustre-MDT0000 complete [10729.916203] LustreError: 376032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852417 with bad export cookie 3918007952388225591 [10729.921182] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10729.927048] LustreError: 376032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10730.321845] Lustre: server umount lustre-MDT0001 complete [10739.447414] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10741.530346] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:27:07 (1788852427) [10749.252034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10749.877208] LustreError: 379654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10749.903589] LustreError: 379654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10749.982556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10755.043550] LustreError: 379655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10755.471657] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10760.164100] LustreError: 379654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.297913] LustreError: 379655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.772649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10766.019183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10770.303976] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10780.336739] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10788.236554] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10796.960575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10797.565559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10806.484581] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10815.223689] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10822.774993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10822.914325] Lustre: lustre-OST0001: new disk, initializing [10822.917184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10822.928657] Lustre: 382493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10823.004120] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10830.188429] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [10832.083892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10832.101250] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10832.285394] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10841.640365] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10847.724147] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10847.726967] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10847.741272] LustreError: Skipped 2 previous similar messages [10847.745072] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10847.783533] Lustre: Skipped 2 previous similar messages [10851.323423] Lustre: server umount lustre-OST0001 complete [10852.837899] LustreError: 381478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10852.851162] LustreError: 381478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10863.083279] LustreError: 383138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10863.090838] LustreError: 383138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10868.194978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10868.216549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10868.220957] Lustre: Skipped 3 previous similar messages [10873.326229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10873.332523] Lustre: Skipped 1 previous similar message [10874.275706] Lustre: server umount lustre-OST0000 complete [10878.952514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10878.967101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10878.983612] Lustre: Skipped 1 previous similar message [10883.553954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10883.555687] Lustre: Skipped 1 previous similar message [10893.279611] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10893.728435] Lustre: server umount lustre-MDT0000 complete [10893.808561] LustreError: 379654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.863410] LustreError: 379654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10897.699315] LustreError: 379637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852585 with bad export cookie 3918007952388226473 [10897.707877] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10897.725775] LustreError: 379637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10904.149492] Lustre: server umount lustre-MDT0001 complete [10915.918616] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [10919.725800] Key type lgssc unregistered [10920.025809] LNet: 384485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10920.038313] LNetError: 384485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10920.057265] LNet: Removed LNI 192.168.202.117@tcp [10921.375563] Key type .llcrypt unregistered [10921.380163] Key type ._llcrypt unregistered [10974.608906] Key type ._llcrypt registered [10974.610146] Key type .llcrypt registered [10974.778444] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [10975.471886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10975.551686] alg: No test for adler32 (adler32-zlib) [10976.919700] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [10977.402930] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [10979.183212] Key type lgssc registered [10981.117840] Lustre: Echo OBD driver; http://www.lustre.org/ [10990.754740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10990.789086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10992.183065] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10992.228815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10992.492330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10997.462769] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11005.344909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11005.448755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11005.475282] Lustre: 387280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11005.782307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11009.063319] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11015.802498] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11022.285367] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11026.402303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11026.429121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11026.441295] Lustre: Skipped 1 previous similar message [11031.331672] Lustre: server umount lustre-MDT0000 complete [11031.522139] LustreError: 387293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11031.544500] LustreError: 387293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11035.230092] LustreError: 387272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852723 with bad export cookie 12633739428846050422 [11035.234394] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11035.238034] LustreError: 387272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11035.449482] Lustre: server umount lustre-MDT0001 complete [11045.032852] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:32:12 (1788852732) [11052.341366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11052.753670] LustreError: 389793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11052.768317] LustreError: 389793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11052.797576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11058.151424] LustreError: 389794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.861681] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11063.266723] LustreError: 389793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.391431] LustreError: 389794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.559122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11068.933241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11073.510169] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11082.093725] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11089.164396] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11098.626852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11099.092763] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11099.454482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11107.006765] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11114.900633] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11122.839662] Lustre: Permanently deactivating lustre-OST0000 [11122.862830] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11129.888525] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11131.017400] LustreError: 392733:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11131.024345] LustreError: 392733:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11131.028856] LustreError: 392733:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11133.402412] LustreError: 392782:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11133.408161] LustreError: 392782:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11141.611476] Lustre: server umount lustre-OST0000 complete [11145.693776] LustreError: 393031:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11145.708405] LustreError: 393031:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11150.817462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11150.838584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11150.852876] Lustre: Skipped 1 previous similar message [11150.859076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11152.872494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.883829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11152.917797] Lustre: Skipped 1 previous similar message [11157.987354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11158.006046] Lustre: Skipped 1 previous similar message [11159.309642] Lustre: server umount lustre-MDT0000 complete [11163.106913] LustreError: 391632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.134324] LustreError: 391632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11164.621698] LustreError: 389774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852852 with bad export cookie 12633739428846051003 [11164.625417] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11164.650211] LustreError: 389774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11175.076052] Lustre: server umount lustre-MDT0001 complete [11185.132667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.681438] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11193.920934] LustreError: 394192:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11201.241900] Lustre: 394531:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.117@tcp [11203.288837] Lustre: server umount MGS complete [11215.208755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11215.788771] LustreError: 395256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11215.809617] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11215.815729] Lustre: Skipped 1 previous similar message [11215.866430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11220.985425] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11230.224712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11230.644963] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11235.091837] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11243.725422] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11250.302356] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11253.147389] Lustre: Permanently reactivating lustre-OST0000 [11253.189727] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11253.205621] Lustre: Skipped 2 previous similar messages [11280.866671] LustreError: 397000:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11289.857986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11290.177567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11290.179988] Lustre: Skipped 1 previous similar message [11297.793976] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11306.836595] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11315.683269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11315.696091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11315.710583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11315.715867] Lustre: Skipped 1 previous similar message [11321.409548] Lustre: server umount lustre-OST0000 complete [11325.921426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11325.929426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11325.958103] Lustre: Skipped 2 previous similar messages [11325.990935] Lustre: Skipped 3 previous similar messages [11332.057331] Lustre: server umount lustre-MDT0000 complete [11335.711570] LustreError: 395238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853023 with bad export cookie 12633739428846052053 [11335.712269] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11335.722484] LustreError: 395238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11336.389658] Lustre: server umount lustre-MDT0001 complete [11347.807232] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [11350.796383] Key type lgssc unregistered [11351.039682] LNet: 399574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11351.050279] LNetError: 399574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11351.072769] LNet: Removed LNI 192.168.202.117@tcp [11351.821434] Key type .llcrypt unregistered [11351.823320] Key type ._llcrypt unregistered [11371.105901] Key type ._llcrypt registered [11371.109856] Key type .llcrypt registered [11371.223827] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [11389.342787] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [11390.454087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11390.655738] alg: No test for adler32 (adler32-zlib) [11391.963807] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [11392.393565] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [11394.103246] Key type lgssc registered [11395.255820] Lustre: Echo OBD driver; http://www.lustre.org/ [11429.177724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11429.190734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.379295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11430.413799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11430.503339] Lustre: lustre-MDT0000: new disk, initializing [11430.592509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11430.605664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11435.521601] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11445.622893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.734314] Lustre: 402729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11445.767729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11445.769637] Lustre: Skipped 1 previous similar message [11445.811144] Lustre: lustre-MDT0001: new disk, initializing [11445.897564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11445.943045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11445.954096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11449.662398] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11460.793161] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11469.443541] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11477.605079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11478.161454] Lustre: lustre-OST0000: new disk, initializing [11478.170930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11478.189088] Lustre: 404701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11478.411725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11486.321644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11486.350580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11486.587089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11487.423842] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11499.606326] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11504.963659] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11505.272780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11510.046383] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11510.260948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11512.291890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.301464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11512.307703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11517.420699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11517.427336] Lustre: Skipped 2 previous similar messages [11518.165028] Lustre: server umount lustre-OST0000 complete [11527.658874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11527.666698] Lustre: Skipped 1 previous similar message [11527.673747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11527.684343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.704198] Lustre: Skipped 1 previous similar message [11529.782648] Lustre: server umount lustre-MDT0000 complete [11533.663347] LustreError: 403526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853221 with bad export cookie 16055899569580501734 [11533.678342] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11533.679969] LustreError: 403526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11534.166505] Lustre: server umount lustre-MDT0001 complete [11542.969772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11544.634456] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:40:31 (1788853231) [11551.875709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11552.261109] LustreError: 407160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11552.290820] LustreError: 407160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11552.438747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11557.182592] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11557.862239] LustreError: 407161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11562.985151] LustreError: 407160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.708965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11568.113567] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11568.549132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11574.587980] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11582.428579] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11588.682874] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11598.033097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11598.495290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11607.024636] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11620.407493] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11624.058283] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11626.562434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11633.632742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11633.639061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11633.651138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11633.654291] Lustre: Skipped 1 previous similar message [11633.670272] Lustre: Skipped 1 previous similar message [11639.680732] Lustre: server umount lustre-OST0000 complete [11643.873330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11643.883371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11643.895952] Lustre: Skipped 1 previous similar message [11643.908020] Lustre: Skipped 3 previous similar messages [11650.044548] Lustre: server umount lustre-MDT0000 complete [11653.707544] LustreError: 407140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853341 with bad export cookie 16055899569580502616 [11653.710373] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11653.724960] LustreError: 407140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11654.194618] Lustre: server umount lustre-MDT0001 complete [11666.272816] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [11669.657608] Key type lgssc unregistered [11669.924137] LNet: 410888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11669.927700] LNetError: 410888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11669.938704] LNet: Removed LNI 192.168.202.117@tcp [11670.844138] Key type .llcrypt unregistered [11670.854179] Key type ._llcrypt unregistered [11699.805243] Key type ._llcrypt registered [11699.811933] Key type .llcrypt registered [11700.006350] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [11700.469675] Key type .llcrypt unregistered [11700.473983] Key type ._llcrypt unregistered [11719.171382] Key type ._llcrypt registered [11719.173684] Key type .llcrypt registered [11719.312418] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [11721.024794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11721.088898] alg: No test for adler32 (adler32-zlib) [11722.238222] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [11722.598079] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [11724.271934] Key type lgssc registered [11725.501588] Lustre: Echo OBD driver; http://www.lustre.org/ [11734.284944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11734.307997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11735.753966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11741.007229] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11745.254256] LustreError: 413104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11745.277671] LustreError: 413104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11749.234306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11749.435913] LustreError: 413103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11749.458420] LustreError: 413103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11749.470591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11754.464965] LustreError: 413098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11754.512840] LustreError: 413098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11756.378974] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11759.600312] LustreError: 413103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11759.611914] LustreError: 413103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11760.642646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11763.242508] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11765.748216] LustreError: 413099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11765.795429] LustreError: 413099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11771.113098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11771.385359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11776.889614] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11786.208250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11786.214626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.246691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11786.720810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11786.740841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.783394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11787.333197] Lustre: server umount lustre-OST0000 complete [11791.850155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11791.851253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11791.866134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11796.963761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11796.966456] Lustre: Skipped 1 previous similar message [11802.080688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11802.103724] Lustre: Skipped 1 previous similar message [11806.175300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11806.326257] Lustre: server umount lustre-MDT0000 complete [11810.407522] LustreError: 413085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853498 with bad export cookie 103775326897395406 [11810.420278] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11810.424557] LustreError: 413085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11810.768175] Lustre: server umount lustre-MDT0001 complete [11821.692807] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [11824.782544] Key type lgssc unregistered [11825.056899] LNet: 416251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11825.062629] LNetError: 416251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11826.088413] LNet: Removed LNI 192.168.202.117@tcp [11826.770646] Key type .llcrypt unregistered [11826.772695] Key type ._llcrypt unregistered [11852.218501] Key type ._llcrypt registered [11852.222935] Key type .llcrypt registered [11852.336788] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [11853.462916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11853.595077] alg: No test for adler32 (adler32-zlib) [11854.681317] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [11854.865685] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [11856.528559] Key type lgssc registered [11857.501984] Lustre: Echo OBD driver; http://www.lustre.org/ [11865.925737] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11865.938822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11881.503928] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11886.623456] LustreError: 417642:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.117@tcp: failed processing log, type 4: rc = -110 [11913.311206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11920.455074] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11927.675324] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11933.905237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11934.132093] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11934.144152] Lustre: MGS: Client 5a04cc1c-61ab-479c-9825-90168bee2f06 (at 0@lo) reconnecting [11934.147355] Lustre: MGC192.168.202.117@tcp: Connection restored to 0@lo (at 0@lo) [11934.621131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11934.735525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11939.819891] LustreError: 417668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11939.835729] LustreError: 417668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11941.304983] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11944.932716] LustreError: 417667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11948.004054] LustreError: 417668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11948.020957] LustreError: 417668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11949.195250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11949.826852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11953.808963] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [11961.520266] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11966.595348] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11980.773163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11980.777797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11980.781073] LustreError: Skipped 1 previous similar message [11980.788862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11980.794669] Lustre: Skipped 1 previous similar message [11980.830085] Lustre: Skipped 1 previous similar message [11985.890624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11985.909730] Lustre: Skipped 1 previous similar message [11989.986204] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11990.329789] Lustre: server umount lustre-OST0000 complete [11996.133275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11996.154385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11996.154885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12000.062695] Lustre: server umount lustre-MDT0000 complete [12001.266573] LustreError: 418685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12001.291891] LustreError: 418685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12004.305169] LustreError: 417650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853692 with bad export cookie 5620279211666856802 [12004.309680] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12004.321230] LustreError: 417650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12004.641253] Lustre: server umount lustre-MDT0001 complete [12017.502340] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [12019.847934] Key type lgssc unregistered [12020.146871] LNet: 421212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12020.151251] LNetError: 421212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12020.177319] LNet: Removed LNI 192.168.202.117@tcp [12021.083091] Key type .llcrypt unregistered [12021.084864] Key type ._llcrypt unregistered [12048.564815] Key type ._llcrypt registered [12048.567655] Key type .llcrypt registered [12048.669277] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [12049.991948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12050.222280] alg: No test for adler32 (adler32-zlib) [12051.528865] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [12052.059124] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [12053.823425] Key type lgssc registered [12055.640434] Lustre: Echo OBD driver; http://www.lustre.org/ [12064.578576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12064.602168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12066.115294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12071.596741] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12075.520585] LustreError: 422628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12075.550616] LustreError: 422628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12079.765752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12080.082255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12084.235435] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12092.332410] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12098.985458] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12105.366918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12105.644455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12110.793746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12110.998401] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12119.536914] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12125.537859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12125.546650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12125.550141] Lustre: Skipped 1 previous similar message [12127.716422] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.17@tcp (stopping) [12130.806403] LustreError: 424028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12130.818872] LustreError: 424028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12130.855060] Lustre: server umount lustre-MDT0000 complete [12132.826535] LustreError: 422623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12133.916675] LustreError: 423365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.17@tcp arrived at 1788853821 with bad export cookie 2063276752419360324 [12133.930568] LustreError: 423365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12135.893884] LustreError: 422610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853823 with bad export cookie 2063276752419359435 [12135.905351] LustreError: 422628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.911176] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12135.913966] LustreError: 422610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12135.917674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.932741] LustreError: 422628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12135.959522] Lustre: Skipped 2 previous similar messages [12135.963460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12141.025263] LustreError: 422629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12141.030614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12141.047883] LustreError: 422629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12142.310688] Lustre: server umount lustre-OST0000 complete [12150.175321] LustreError: 423359:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000019af8148 x1875749098258176/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12150.414448] Lustre: server umount lustre-MDT0001 complete [12161.596702] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [12165.004156] Key type lgssc unregistered [12165.322167] LNet: 426313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12165.329868] LNetError: 426313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12165.352638] LNet: Removed LNI 192.168.202.117@tcp [12166.873212] Key type .llcrypt unregistered [12166.874993] Key type ._llcrypt unregistered [12194.035372] Key type ._llcrypt registered [12194.038124] Key type .llcrypt registered [12194.140814] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [12195.168893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12195.366196] alg: No test for adler32 (adler32-zlib) [12196.517466] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [12196.802982] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [12198.479151] Key type lgssc registered [12199.531294] Lustre: Echo OBD driver; http://www.lustre.org/ [12206.712202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12206.725723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12208.214302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12212.713688] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12217.331116] LustreError: 427728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12217.353897] LustreError: 427728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12222.436172] LustreError: 427729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12224.491599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12225.349954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12231.544786] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12240.087901] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12246.377409] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12253.424140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12253.808379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12260.469531] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12268.028803] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12277.363531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12282.850740] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12282.857776] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12282.865979] Lustre: Skipped 1 previous similar message [12284.920598] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.17@tcp (stopping) [12287.968907] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12287.988746] Lustre: Skipped 1 previous similar message [12288.684160] Lustre: server umount lustre-MDT0001 complete [12290.005268] LustreError: 427724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12293.122882] LustreError: 427724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12293.159583] LustreError: 427724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12298.208892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12298.213697] LustreError: 427728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12298.226126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12298.226134] Lustre: Skipped 1 previous similar message [12298.228338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12300.195692] Lustre: server umount lustre-OST0000 complete [12304.546836] Lustre: server umount lustre-MDT0000 complete [12319.138049] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [12321.991781] Key type lgssc unregistered [12322.375741] LNet: 431434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12322.386786] LNetError: 431434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12323.433366] LNet: Removed LNI 192.168.202.117@tcp [12324.795285] Key type .llcrypt unregistered [12324.797619] Key type ._llcrypt unregistered [12354.633254] Key type ._llcrypt registered [12354.642745] Key type .llcrypt registered [12354.854799] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [12373.947487] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [12375.169832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12375.178153] alg: No test for adler32 (adler32-zlib) [12376.372240] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [12376.825952] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [12378.524678] Key type lgssc registered [12379.961978] Lustre: Echo OBD driver; http://www.lustre.org/ [12419.549765] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [12438.402874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12438.448727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12439.790453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12439.851989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12439.986843] Lustre: lustre-MDT0000: new disk, initializing [12440.095423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12440.109913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12445.297900] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12459.541968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12459.788717] Lustre: lustre-OST0000: new disk, initializing [12459.792462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12459.796545] Lustre: Skipped 1 previous similar message [12459.800782] Lustre: 436468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12459.863370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12465.804805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12465.825501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12465.842560] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12465.869953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12473.872515] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12486.748308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12486.878871] Lustre: 435448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12486.971374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12487.036753] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.17@tcp (not set up) [12487.047133] Lustre: lustre-MDT0001: new disk, initializing [12487.131860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12487.171477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12487.178777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12492.992740] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12503.227596] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12503.634857] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12508.560507] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12508.866524] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12516.322850] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12516.332059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12516.342201] Lustre: Skipped 1 previous similar message [12516.364537] Lustre: Skipped 1 previous similar message [12517.861736] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.17@tcp (stopping) [12521.712917] Lustre: server umount lustre-MDT0001 complete [12526.638971] LustreError: 435456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12526.654989] LustreError: 435456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12528.091074] LustreError: 438007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.685105] LustreError: 435455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.704867] LustreError: 435455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12536.812579] LustreError: 435461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12536.845084] LustreError: 435461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12541.923260] LustreError: 435456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12541.939982] LustreError: 435456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12542.944778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12542.949737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12542.961083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12547.042325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12548.653832] Lustre: server umount lustre-OST0000 complete [12558.436842] Lustre: server umount lustre-MDT0000 complete [12570.722769] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [12573.657213] Key type lgssc unregistered [12573.966545] LNet: 440216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12573.988731] LNetError: 440216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12574.014942] LNet: Removed LNI 192.168.202.117@tcp [12574.965083] Key type .llcrypt unregistered [12574.966790] Key type ._llcrypt unregistered [12615.768043] Key type ._llcrypt registered [12615.770356] Key type .llcrypt registered [12615.885390] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [12616.166669] Key type .llcrypt unregistered [12616.168083] Key type ._llcrypt unregistered [12632.265293] Key type ._llcrypt registered [12632.267732] Key type .llcrypt registered [12632.362269] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [12633.527943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12633.584446] alg: No test for adler32 (adler32-zlib) [12634.599795] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [12634.773666] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [12636.399324] Key type lgssc registered [12637.809359] Lustre: Echo OBD driver; http://www.lustre.org/ [12668.031941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12668.059523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12669.276125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12669.300002] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12669.382581] Lustre: lustre-MDT0000: new disk, initializing [12669.525908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12669.536474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12675.528965] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12686.132699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12686.234250] Lustre: 443589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12686.315426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12686.320666] Lustre: Skipped 1 previous similar message [12686.421521] Lustre: lustre-MDT0001: new disk, initializing [12686.514815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12686.564072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12686.578975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12691.291420] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12700.281276] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12705.313062] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12712.166851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12712.412882] Lustre: lustre-OST0000: new disk, initializing [12712.419085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12712.428282] Lustre: 445558:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12712.526323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12718.361236] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12719.199778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12719.215610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12719.349717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12728.530968] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12738.017979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12738.020733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12738.034041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12740.065124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12740.084320] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12740.101007] Lustre: Skipped 1 previous similar message [12742.379558] Lustre: server umount lustre-MDT0000 complete [12747.178105] LustreError: 446279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854435 with bad export cookie 2599565573466095888 [12747.202108] LustreError: 446279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12747.210567] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12747.718818] Lustre: server umount lustre-MDT0001 complete [12762.848992] Lustre: server umount lustre-OST0000 complete [12775.355073] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:01:01 (1788854461) [12783.691180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12784.000104] LustreError: 447597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12784.011469] LustreError: 447597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12784.061039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12787.758705] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12789.217323] LustreError: 447598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12789.258566] LustreError: 447598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12793.317650] LustreError: 447597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12795.375421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12795.848170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12800.672755] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12810.566605] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12817.045140] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12838.892540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12839.134914] Lustre: Found index 0 for lustre-OST0000, updating log [12839.470498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12848.960219] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [12855.859682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12859.269398] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12866.025694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12866.037627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12866.050385] Lustre: Skipped 1 previous similar message [12866.060578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12866.068667] Lustre: Skipped 3 previous similar messages [12869.691512] Lustre: server umount lustre-OST0000 complete [12888.032209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12888.051697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12888.084114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12888.098032] Lustre: Skipped 2 previous similar messages [12893.416338] Lustre: server umount lustre-MDT0000 complete [12893.665634] LustreError: 447598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12893.691379] LustreError: 447598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12896.436545] LustreError: 447578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854584 with bad export cookie 2599565573466097281 [12896.470365] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12896.814292] Lustre: server umount lustre-MDT0001 complete [12899.049714] LNet: 1 local NIs in recovery (showing 1): 192.168.202.117@tcp [12907.424279] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:03:14 (1788854594) [13535.663896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13536.454867] LustreError: 454011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13536.477975] LustreError: 454011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13536.545486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13541.861928] LustreError: 454012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13542.707812] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [13546.977683] LustreError: 454011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13552.925142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13553.122081] LustreError: 454012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13553.160374] LustreError: 454012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13553.724105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13559.155926] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [13567.743716] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13577.084995] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13589.345060] LNet: 1 local NIs in recovery (showing 1): 192.168.202.117@tcp [13600.298930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13600.604865] Lustre: Found index 0 for lustre-OST0000, updating log [13600.955236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13608.578600] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [13617.448171] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13620.852621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13620.876649] Lustre: Skipped 1 previous similar message [13620.889713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13620.902028] Lustre: Skipped 1 previous similar message [13626.099141] Lustre: server umount lustre-OST0000 complete [13630.432459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13630.438967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13630.459580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13631.520864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13631.530568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13633.644984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13633.664718] Lustre: Skipped 4 previous similar messages [13637.731622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13637.744631] Lustre: Skipped 7 previous similar messages [13644.255148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13644.415955] Lustre: server umount lustre-MDT0000 complete [13644.903904] LustreError: 454011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13644.928814] LustreError: 454011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13648.551062] LustreError: 453991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855336 with bad export cookie 2599565573466098366 [13648.561963] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13648.562662] LustreError: 453991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13648.958486] Lustre: server umount lustre-MDT0001 complete [13651.818783] LNet: 1 local NIs in recovery (showing 1): 192.168.202.117@tcp [13686.381437] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [13696.414206] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [13740.339772] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:17:06 (1788855426) [13760.966698] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [13764.569204] Key type lgssc unregistered [13764.984834] LNet: 460630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13764.995554] LNetError: 460630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13862.313656] LNet: Removed LNI 192.168.202.117@tcp [13863.455685] Key type .llcrypt unregistered [13863.458420] Key type ._llcrypt unregistered [13880.470146] Key type ._llcrypt registered [13880.471811] Key type .llcrypt registered [13880.592573] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [13882.057582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13882.113825] alg: No test for adler32 (adler32-zlib) [13883.235965] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [13883.641728] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [13885.447185] Key type lgssc registered [13887.888266] Lustre: Echo OBD driver; http://www.lustre.org/ [13893.035479] LNet: Added LNI 192.168.202.117@tcp100 [8/256/0/180] [13893.942577] LNet: Added LNI 192.168.202.117@tcp101 [8/256/0/180] [13894.876517] LNet: Added LNI 192.168.202.117@tcp102 [8/256/0/180] [13895.833746] LNet: Added LNI 192.168.202.117@tcp103 [8/256/0/180] [13896.832218] LNet: Added LNI 192.168.202.117@tcp104 [8/256/0/180] [13897.813367] LNet: Added LNI 192.168.202.117@tcp105 [8/256/0/180] [13898.697821] LNet: Added LNI 192.168.202.117@tcp106 [8/256/0/180] [13899.493561] LNet: Added LNI 192.168.202.117@tcp107 [8/256/0/180] [13900.371092] LNet: Added LNI 192.168.202.117@tcp108 [8/256/0/180] [13901.348884] LNet: Added LNI 192.168.202.117@tcp109 [8/256/0/180] [13902.249718] LNet: Added LNI 192.168.202.117@tcp110 [8/256/0/180] [13903.193661] LNet: Added LNI 192.168.202.117@tcp111 [8/256/0/180] [13904.268443] LNet: Added LNI 192.168.202.117@tcp112 [8/256/0/180] [13905.169611] LNet: Added LNI 192.168.202.117@tcp113 [8/256/0/180] [13906.113792] LNet: Added LNI 192.168.202.117@tcp114 [8/256/0/180] [13906.839951] LNet: Added LNI 192.168.202.117@tcp115 [8/256/0/180] [13907.994095] LNet: Added LNI 192.168.202.117@tcp116 [8/256/0/180] [13909.028962] LNet: Added LNI 192.168.202.117@tcp117 [8/256/0/180] [13910.058228] LNet: Added LNI 192.168.202.117@tcp118 [8/256/0/180] [13911.087458] LNet: Added LNI 192.168.202.117@tcp119 [8/256/0/180] [13911.913571] LNet: Added LNI 192.168.202.117@tcp120 [8/256/0/180] [13912.761037] LNet: Added LNI 192.168.202.117@tcp121 [8/256/0/180] [13913.713924] LNet: Added LNI 192.168.202.117@tcp122 [8/256/0/180] [13914.650993] LNet: Added LNI 192.168.202.117@tcp123 [8/256/0/180] [13915.568658] LNet: Added LNI 192.168.202.117@tcp124 [8/256/0/180] [13916.716986] LNet: Added LNI 192.168.202.117@tcp125 [8/256/0/180] [13917.698854] LNet: Added LNI 192.168.202.117@tcp126 [8/256/0/180] [13918.803246] LNet: Added LNI 192.168.202.117@tcp127 [8/256/0/180] [13919.730587] LNet: Added LNI 192.168.202.117@tcp128 [8/256/0/180] [13920.812982] LNet: Added LNI 192.168.202.117@tcp129 [8/256/0/180] [13921.865023] LNet: Added LNI 192.168.202.117@tcp130 [8/256/0/180] [13922.911032] LNet: Added LNI 192.168.202.117@tcp131 [8/256/0/180] [13924.011897] LNet: Added LNI 192.168.202.117@tcp132 [8/256/0/180] [13925.123058] LNet: Added LNI 192.168.202.117@tcp133 [8/256/0/180] [13926.207786] LNet: Added LNI 192.168.202.117@tcp134 [8/256/0/180] [13927.107551] LNet: Added LNI 192.168.202.117@tcp135 [8/256/0/180] [13934.527425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13934.570865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13935.779530] Lustre: 463656:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.117@tcp: can't add NID 192.168.202.117@tcp130: rc = -75 [13935.871078] LNetError: 461115:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.117 [13935.897491] LNetError: Protocol error connecting to 192.168.202.117@tcp116 at host 192.168.202.117:988: is it running a compatible version of Lustre? [13935.912978] LNetError: Skipped 2 previous similar messages [13935.929137] LNet: 5 local NIs in recovery (showing 5): 192.168.202.117@tcp126, 192.168.202.117@tcp125, 192.168.202.117@tcp129, 192.168.202.117@tcp128, 192.168.202.117@tcp127 [13936.024482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13936.052356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13936.177315] Lustre: lustre-MDT0000: new disk, initializing [13936.239442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13936.264509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13936.928925] LNetError: 461115:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.117 [13936.943036] LNetError: 461115:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [13936.956456] LNetError: Protocol error connecting to 192.168.202.117@tcp114 at host 192.168.202.117:988: is it running a compatible version of Lustre? [13936.962312] LNetError: Skipped 8 previous similar messages [13938.019286] LNetError: 463686:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.117 [13938.039839] LNetError: 463686:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [13938.062566] LNetError: Protocol error connecting to 192.168.202.117@tcp126 at host 192.168.202.117:988: is it running a compatible version of Lustre? [13938.075993] LNetError: Skipped 4 previous similar messages [13941.136595] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [13942.126961] LNetError: 461114:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.117 [13942.143435] LNetError: Protocol error connecting to 192.168.202.117@tcp126 at host 192.168.202.117:988: is it running a compatible version of Lustre? [13942.155643] LNetError: Skipped 1 previous similar message [13950.317687] LNetError: 463687:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.117 [13950.347827] LNetError: 463687:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13950.365428] LNetError: Protocol error connecting to 192.168.202.117@tcp126 at host 192.168.202.117:988: is it running a compatible version of Lustre? [13951.647811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13951.948786] Lustre: 463674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13952.114481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13952.122087] Lustre: Skipped 1 previous similar message [13952.246799] Lustre: lustre-MDT0001: new disk, initializing [13952.295437] Lustre: 464476:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.117@tcp: can't add NID 192.168.202.117@tcp130: rc = -75 [13952.303612] Lustre: 464476:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13952.478389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13952.559768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13952.571225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13958.121726] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [13965.605600] LNetError: 461115:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.117 [13965.615801] LNetError: Protocol error connecting to 192.168.202.117@tcp116 at host 192.168.202.117:988: is it running a compatible version of Lustre? [13968.664541] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13975.022506] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13981.565397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13981.967600] Lustre: lustre-OST0000: new disk, initializing [13981.977601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13981.981591] Lustre: 465647:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13981.984319] LNetError: 461115:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.117 [13982.014175] LNetError: 461115:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [13982.025956] LNetError: Protocol error connecting to 192.168.202.117@tcp127 at host 192.168.202.117:988: is it running a compatible version of Lustre? [13982.035317] LNetError: Skipped 10 previous similar messages [13982.080289] Lustre: 465652:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.117@tcp: can't add NID 192.168.202.117@tcp130: rc = -75 [13982.085314] Lustre: 465652:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13982.129897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13989.283757] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [13989.937970] Lustre: 466027:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.117@tcp: can't add NID 192.168.202.117@tcp130: rc = -75 [13989.960160] Lustre: 466027:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13990.032526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13990.056900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13990.205848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13995.359300] LNet: 1 local NIs in recovery (showing 1): 192.168.202.117@tcp126 [13998.888361] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14010.853575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14010.875833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14010.903548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14010.907106] Lustre: Skipped 1 previous similar message [14012.555915] Lustre: server umount lustre-OST0000 complete [14015.776390] LNetError: 461115:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.117 [14015.782612] LNetError: 461115:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 21 previous similar messages [14015.787510] LNetError: Protocol error connecting to 192.168.202.117@tcp121 at host 192.168.202.117:988: is it running a compatible version of Lustre? [14015.802342] LNetError: Skipped 21 previous similar messages [14019.041038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14019.046128] LustreError: Skipped 1 previous similar message [14019.049726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14019.057252] Lustre: Skipped 1 previous similar message [14019.064218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14021.089449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14021.092855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14026.213121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14026.225149] Lustre: Skipped 2 previous similar messages [14030.306348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14030.315704] Lustre: Skipped 1 previous similar message [14031.327545] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14031.882669] Lustre: server umount lustre-MDT0000 complete [14037.179903] LustreError: 463664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855725 with bad export cookie 16427650498898975943 [14037.192186] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14037.195482] LustreError: 463664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14037.511521] Lustre: server umount lustre-MDT0001 complete [14047.958206] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [14051.503956] Key type lgssc unregistered [14051.908458] LNet: 467565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14051.913277] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14051.920503] LNet: Removed LNI 192.168.202.117@tcp [14052.959172] LNet: Removed LNI 192.168.202.117@tcp100 [14052.969184] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14052.985471] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14052.998271] LNet: Removed LNI 192.168.202.117@tcp101 [14054.047266] LNet: Removed LNI 192.168.202.117@tcp102 [14054.060495] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14054.074603] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14054.105300] LNet: Removed LNI 192.168.202.117@tcp103 [14055.135484] LNet: Removed LNI 192.168.202.117@tcp104 [14056.159615] LNet: Removed LNI 192.168.202.117@tcp105 [14056.166324] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14056.170531] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14056.180681] LNet: Removed LNI 192.168.202.117@tcp106 [14057.249336] LNet: Removed LNI 192.168.202.117@tcp107 [14058.271472] LNet: Removed LNI 192.168.202.117@tcp108 [14059.295984] LNet: Removed LNI 192.168.202.117@tcp109 [14060.319164] LNet: Removed LNI 192.168.202.117@tcp110 [14060.323373] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14060.335233] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14060.350706] LNet: Removed LNI 192.168.202.117@tcp111 [14061.407880] LNet: Removed LNI 192.168.202.117@tcp112 [14062.431139] LNet: Removed LNI 192.168.202.117@tcp113 [14063.456059] LNet: Removed LNI 192.168.202.117@tcp114 [14064.479546] LNet: Removed LNI 192.168.202.117@tcp115 [14065.503161] LNet: Removed LNI 192.168.202.117@tcp116 [14066.591401] LNet: Removed LNI 192.168.202.117@tcp117 [14067.616630] LNet: Removed LNI 192.168.202.117@tcp118 [14068.639710] LNet: Removed LNI 192.168.202.117@tcp119 [14068.650608] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14068.658243] LNetError: 467565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14068.694483] LNet: Removed LNI 192.168.202.117@tcp120 [14069.727214] LNet: Removed LNI 192.168.202.117@tcp121 [14070.752615] LNet: Removed LNI 192.168.202.117@tcp122 [14071.776096] LNet: Removed LNI 192.168.202.117@tcp123 [14072.799195] LNet: Removed LNI 192.168.202.117@tcp124 [14073.823156] LNet: Removed LNI 192.168.202.117@tcp125 [14073.825253] LNet: Removed LNI 192.168.202.117@tcp126 [14074.912053] LNet: Removed LNI 192.168.202.117@tcp127 [14075.938662] LNet: Removed LNI 192.168.202.117@tcp128 [14076.959289] LNet: Removed LNI 192.168.202.117@tcp129 [14076.966708] LNet: Removed LNI 192.168.202.117@tcp130 [14076.977515] LNet: Removed LNI 192.168.202.117@tcp131 [14076.984063] LNet: Removed LNI 192.168.202.117@tcp132 [14076.988045] LNet: Removed LNI 192.168.202.117@tcp133 [14076.998619] LNet: Removed LNI 192.168.202.117@tcp134 [14077.015292] LNet: Removed LNI 192.168.202.117@tcp135 [14077.982690] Key type .llcrypt unregistered [14077.989234] Key type ._llcrypt unregistered [14103.980948] Key type ._llcrypt registered [14103.995314] Key type .llcrypt registered [14104.213181] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [14122.269857] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [14123.566783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14123.599293] alg: No test for adler32 (adler32-zlib) [14124.731414] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [14124.967147] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [14126.679198] Key type lgssc registered [14127.985694] Lustre: Echo OBD driver; http://www.lustre.org/ [14162.278300] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:24:09 (1788855849) [14183.778919] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [14186.968290] Key type lgssc unregistered [14187.445112] LNet: 471652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14187.452572] LNetError: 471652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14187.487018] LNet: Removed LNI 192.168.202.117@tcp [14188.684425] Key type .llcrypt unregistered [14188.688098] Key type ._llcrypt unregistered [14207.733175] Key type ._llcrypt registered [14207.734829] Key type .llcrypt registered [14207.834375] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [14209.644128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14209.691928] alg: No test for adler32 (adler32-zlib) [14210.883977] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [14211.249351] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [14213.031364] Key type lgssc registered [14214.333560] Lustre: Echo OBD driver; http://www.lustre.org/ [14224.672310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14224.682321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14226.003360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14226.026401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14226.161837] Lustre: lustre-MDT0000: new disk, initializing [14226.235298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14226.300586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14230.829030] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [14241.655688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14241.855637] Lustre: 472860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14241.989867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14242.000811] Lustre: Skipped 1 previous similar message [14242.113607] Lustre: lustre-MDT0001: new disk, initializing [14242.181113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14242.213831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14242.234570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14248.218696] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [14259.553737] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14265.764460] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14272.478938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14272.759342] Lustre: lustre-OST0000: new disk, initializing [14272.762557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14272.768465] Lustre: 474831:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14272.869722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14277.761117] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [14278.753428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14278.766457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14278.881089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14286.653541] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14293.059827] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing lnet_if_list [14300.142215] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14300.160364] LNet: Added LNI 192.168.202.117@tcp10 [8/256/0/180] [14309.283585] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14309.351439] LNet: Added LNI 192.168.202.117@tcp11 [8/256/0/180] [14315.113905] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14315.162244] LNet: Added LNI 192.168.202.117@tcp12 [8/256/0/180] [14319.594677] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14319.619703] LNet: Added LNI 192.168.202.117@tcp13 [8/256/0/180] [14324.811752] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14324.832251] LNet: Added LNI 192.168.202.117@tcp14 [8/256/0/180] [14331.214428] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14331.251802] LNet: Added LNI 192.168.202.117@tcp15 [8/256/0/180] [14338.252473] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14338.287170] LNet: Added LNI 192.168.202.117@tcp16 [8/256/0/180] [14345.656345] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14345.733262] LNet: Added LNI 192.168.202.117@tcp17 [8/256/0/180] [14351.587247] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14351.635564] LNet: Added LNI 192.168.202.117@tcp18 [8/256/0/180] [14359.683835] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14359.739945] LNet: Added LNI 192.168.202.117@tcp19 [8/256/0/180] [14365.228780] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14365.253996] LNet: Added LNI 192.168.202.117@tcp20 [8/256/0/180] [14371.114280] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14371.143761] LNet: Added LNI 192.168.202.117@tcp21 [8/256/0/180] [14377.281252] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14377.351792] LNet: Added LNI 192.168.202.117@tcp22 [8/256/0/180] [14383.559427] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14383.584589] LNet: Added LNI 192.168.202.117@tcp23 [8/256/0/180] [14389.916452] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14389.980767] LNet: Added LNI 192.168.202.117@tcp24 [8/256/0/180] [14396.517805] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14396.549373] LNet: Added LNI 192.168.202.117@tcp25 [8/256/0/180] [14403.207241] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14403.231277] LNet: Added LNI 192.168.202.117@tcp26 [8/256/0/180] [14410.259991] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14410.292903] LNet: Added LNI 192.168.202.117@tcp27 [8/256/0/180] [14417.994607] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14418.029531] LNet: Added LNI 192.168.202.117@tcp28 [8/256/0/180] [14424.722230] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14424.757405] LNet: Added LNI 192.168.202.117@tcp29 [8/256/0/180] [14432.570860] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14432.613582] LNet: Added LNI 192.168.202.117@tcp30 [8/256/0/180] [14440.458639] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14440.488548] LNet: Added LNI 192.168.202.117@tcp31 [8/256/0/180] [14446.922662] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14446.956327] LNet: Added LNI 192.168.202.117@tcp32 [8/256/0/180] [14453.073043] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14453.124813] LNet: Added LNI 192.168.202.117@tcp33 [8/256/0/180] [14461.335868] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14461.391648] LNet: Added LNI 192.168.202.117@tcp34 [8/256/0/180] [14470.470446] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14470.490754] LNet: Added LNI 192.168.202.117@tcp35 [8/256/0/180] [14479.013812] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14479.040696] LNet: Added LNI 192.168.202.117@tcp36 [8/256/0/180] [14486.454267] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14486.479312] LNet: Added LNI 192.168.202.117@tcp37 [8/256/0/180] [14497.082567] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14497.118191] LNet: Added LNI 192.168.202.117@tcp38 [8/256/0/180] [14506.265370] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14506.297497] LNet: Added LNI 192.168.202.117@tcp39 [8/256/0/180] [14515.098454] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14515.133966] LNet: Added LNI 192.168.202.117@tcp40 [8/256/0/180] [14523.323430] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14523.346669] LNet: Added LNI 192.168.202.117@tcp41 [8/256/0/180] [14532.537767] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14532.594836] LNet: Added LNI 192.168.202.117@tcp42 [8/256/0/180] [14548.267080] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14548.349259] LNet: Added LNI 192.168.202.117@tcp43 [8/256/0/180] [14561.741306] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14561.797883] LNet: Added LNI 192.168.202.117@tcp44 [8/256/0/180] [14571.639972] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14571.685361] LNet: Added LNI 192.168.202.117@tcp45 [8/256/0/180] [14581.807509] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14581.862281] LNet: Added LNI 192.168.202.117@tcp46 [8/256/0/180] [14591.462437] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14591.522524] LNet: Added LNI 192.168.202.117@tcp47 [8/256/0/180] [14603.323712] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14603.360773] LNet: Added LNI 192.168.202.117@tcp48 [8/256/0/180] [14613.439202] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14613.472936] LNet: Added LNI 192.168.202.117@tcp49 [8/256/0/180] [14623.620460] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14623.653585] LNet: Added LNI 192.168.202.117@tcp50 [8/256/0/180] [14646.752396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14646.779831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14646.810327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14647.778259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14647.784206] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14652.901737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14652.906243] Lustre: Skipped 2 previous similar messages [14658.026519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14658.034993] Lustre: Skipped 3 previous similar messages [14660.063122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14660.474097] Lustre: server umount lustre-MDT0000 complete [14664.400721] LustreError: 472852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856352 with bad export cookie 4797840008047427305 [14664.409195] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14664.416927] LustreError: 472852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14664.680309] Lustre: server umount lustre-MDT0001 complete [14684.639310] Lustre: 472132:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788856356/real 1788856356] req@000000006687f495 x1875751363345152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788856372 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14684.695755] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14684.715548] Lustre: Skipped 1 previous similar message [14685.192712] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [14685.669217] Lustre: 472134:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788856357/real 1788856357] req@000000006f8fd8f7 x1875751363345280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788856373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14689.759137] Lustre: 472135:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788856361/real 1788856361] req@00000000b1c860f8 x1875751363345408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788856377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14694.137675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14695.967872] Lustre: 472132:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788856367/real 1788856367] req@0000000063707d16 x1875751363345792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788856383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14695.994441] Lustre: 472132:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14701.024798] Lustre: 472135:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788856372/real 1788856372] req@000000008862bfe8 x1875751363345920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788856388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14709.747124] LustreError: 474825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14709.764932] LustreError: 474825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14709.819520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14714.856214] LustreError: 474826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14716.330203] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [14718.958031] LustreError: 474825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14718.964432] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14724.064844] LustreError: 474826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14725.382565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14726.162913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14729.198707] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14732.165648] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [14740.085607] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14746.871846] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14770.149721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14770.152042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14770.183640] Lustre: Skipped 1 previous similar message [14770.204630] Lustre: Skipped 1 previous similar message [14774.528327] Lustre: server umount lustre-OST0000 complete [14780.386615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14780.391753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14780.427599] Lustre: Skipped 1 previous similar message [14780.451764] Lustre: Skipped 1 previous similar message [14786.992957] Lustre: server umount lustre-MDT0000 complete [14790.626794] LustreError: 499084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14790.657899] LustreError: 499084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14791.618681] LustreError: 472852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856479 with bad export cookie 4797840008047429979 [14791.626794] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14791.638677] LustreError: 472852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14792.204632] Lustre: server umount lustre-MDT0001 complete [14805.932205] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [14809.669484] Key type lgssc unregistered [14810.097764] LNet: 501883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14810.103242] LNetError: 501883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14810.109826] LNet: Removed LNI 192.168.202.117@tcp [14810.112443] LNet: Removed LNI 192.168.202.117@tcp10 [14810.115394] LNet: Removed LNI 192.168.202.117@tcp11 [14810.119376] LNet: Removed LNI 192.168.202.117@tcp12 [14810.122962] LNet: Removed LNI 192.168.202.117@tcp13 [14810.126334] LNet: Removed LNI 192.168.202.117@tcp14 [14810.129163] LNet: Removed LNI 192.168.202.117@tcp15 [14810.131750] LNet: Removed LNI 192.168.202.117@tcp16 [14810.134359] LNet: Removed LNI 192.168.202.117@tcp17 [14810.136913] LNet: Removed LNI 192.168.202.117@tcp18 [14810.139178] LNet: Removed LNI 192.168.202.117@tcp19 [14810.141417] LNet: Removed LNI 192.168.202.117@tcp20 [14810.143946] LNet: Removed LNI 192.168.202.117@tcp21 [14810.147282] LNet: Removed LNI 192.168.202.117@tcp22 [14810.150587] LNet: Removed LNI 192.168.202.117@tcp23 [14810.153248] LNet: Removed LNI 192.168.202.117@tcp24 [14810.156047] LNet: Removed LNI 192.168.202.117@tcp25 [14810.159334] LNet: Removed LNI 192.168.202.117@tcp26 [14810.161906] LNet: Removed LNI 192.168.202.117@tcp27 [14810.164458] LNet: Removed LNI 192.168.202.117@tcp28 [14810.167308] LNet: Removed LNI 192.168.202.117@tcp29 [14810.170209] LNet: Removed LNI 192.168.202.117@tcp30 [14810.173356] LNet: Removed LNI 192.168.202.117@tcp31 [14810.176261] LNet: Removed LNI 192.168.202.117@tcp32 [14810.179512] LNet: Removed LNI 192.168.202.117@tcp33 [14810.182215] LNet: Removed LNI 192.168.202.117@tcp34 [14810.185405] LNet: Removed LNI 192.168.202.117@tcp35 [14810.188368] LNet: Removed LNI 192.168.202.117@tcp36 [14810.190881] LNet: Removed LNI 192.168.202.117@tcp37 [14810.193399] LNet: Removed LNI 192.168.202.117@tcp38 [14810.195745] LNet: Removed LNI 192.168.202.117@tcp39 [14810.201545] LNet: Removed LNI 192.168.202.117@tcp40 [14810.204324] LNet: Removed LNI 192.168.202.117@tcp41 [14810.206912] LNet: Removed LNI 192.168.202.117@tcp42 [14810.209598] LNet: Removed LNI 192.168.202.117@tcp43 [14810.216114] LNet: Removed LNI 192.168.202.117@tcp44 [14810.218439] LNet: Removed LNI 192.168.202.117@tcp45 [14810.221290] LNet: Removed LNI 192.168.202.117@tcp46 [14810.226741] LNet: Removed LNI 192.168.202.117@tcp47 [14810.229612] LNet: Removed LNI 192.168.202.117@tcp48 [14810.235726] LNet: Removed LNI 192.168.202.117@tcp49 [14810.256284] LNet: Removed LNI 192.168.202.117@tcp50 [14811.343355] Key type .llcrypt unregistered [14811.345090] Key type ._llcrypt unregistered [14834.579421] Key type ._llcrypt registered [14834.587071] Key type .llcrypt registered [14834.753860] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [14853.562508] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [14855.060360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14855.142465] alg: No test for adler32 (adler32-zlib) [14856.350531] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [14856.626646] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [14858.335214] Key type lgssc registered [14859.826808] Lustre: Echo OBD driver; http://www.lustre.org/ [14905.064849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14905.089384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14906.468049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14906.526790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14906.608759] Lustre: lustre-MDT0000: new disk, initializing [14906.701929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14906.713760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14911.741835] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [14922.434855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14922.578291] Lustre: 505078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14922.663535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14922.665486] Lustre: Skipped 1 previous similar message [14922.772584] Lustre: lustre-MDT0001: new disk, initializing [14922.935986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14922.975391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14922.989537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14928.211826] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [14939.672555] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14945.961352] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14954.291866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14954.588361] Lustre: *** cfs_fail_loc=90f, val=0*** [14954.737279] Lustre: lustre-OST0000: new disk, initializing [14954.751256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14954.770192] Lustre: 507096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14954.950316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14961.366046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14961.397930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14961.620492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14964.317323] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [14977.018973] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14987.236423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14987.239359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14987.244983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14987.256588] Lustre: Skipped 1 previous similar message [14987.280842] Lustre: Skipped 1 previous similar message [14990.942068] Lustre: server umount lustre-OST0000 complete [14997.472236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14997.490068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14997.491889] Lustre: Skipped 1 previous similar message [14997.520058] Lustre: Skipped 1 previous similar message [15002.197922] Lustre: server umount lustre-MDT0000 complete [15006.281180] LustreError: 505070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856694 with bad export cookie 11852842507075033924 [15006.283157] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15006.291198] LustreError: 505070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15006.978568] Lustre: server umount lustre-MDT0001 complete [15021.329637] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [15025.207408] Key type lgssc unregistered [15025.622201] LNet: 509003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15025.635560] LNetError: 509003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15025.663573] LNet: Removed LNI 192.168.202.117@tcp [15026.640254] Key type .llcrypt unregistered [15026.642220] Key type ._llcrypt unregistered [15048.419900] Key type ._llcrypt registered [15048.423573] Key type .llcrypt registered [15048.578887] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [15049.758461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15049.976047] alg: No test for adler32 (adler32-zlib) [15051.127424] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [15051.487422] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [15053.271305] Key type lgssc registered [15054.595943] Lustre: Echo OBD driver; http://www.lustre.org/ [15064.557238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15064.586361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15066.721063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15072.085836] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [15076.387596] LustreError: 510268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15076.454514] LustreError: 510268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15081.448844] LustreError: 510267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15084.553199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15084.863504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15090.697802] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [15099.214665] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15105.682432] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15112.885292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15113.297569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15121.440093] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [15129.517964] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15150.730405] loop0: detected capacity change from 0 to 204800000 [15150.981223] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15150.996687] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15151.005122] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15158.682812] loop0: detected capacity change from 0 to 204800000 [15158.850623] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15158.857857] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15158.878827] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15164.038148] loop0: detected capacity change from 0 to 204800000 [15169.872906] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15170.068369] Lustre: fs15246-OST0000: new disk, initializing [15170.070821] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15170.080926] Lustre: 513883:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15170.154634] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15176.739540] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [15187.006602] loop1: detected capacity change from 0 to 204800000 [15193.576526] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15193.679519] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15193.763920] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15193.865299] Lustre: fs15246-MDT0000: new disk, initializing [15194.013273] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15194.043298] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15194.153417] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15195.172058] Lustre: 515005:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000035b421a1 x1875752244006784/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788856898 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15198.094342] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [15203.367288] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15203.385606] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15203.408995] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15222.267128] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15222.297217] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15226.176632] Lustre: server umount fs15246-MDT0000 complete [15227.370540] LustreError: 510263:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15232.489259] LustreError: 512556:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15237.627368] LustreError: 510262:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15239.713334] Lustre: server umount fs15246-OST0000 complete [15258.087461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15258.089732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15258.093489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15258.109986] Lustre: Skipped 1 previous similar message [15263.215853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15263.218739] Lustre: Skipped 1 previous similar message [15263.965770] Lustre: server umount lustre-OST0000 complete [15268.328343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15268.332904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15268.335484] Lustre: Skipped 1 previous similar message [15273.453744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15273.478861] Lustre: Skipped 2 previous similar messages [15274.724826] Lustre: server umount lustre-MDT0000 complete [15278.561109] LustreError: 510267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15278.574501] LustreError: 510267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15279.281774] LustreError: 510249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856967 with bad export cookie 13007443809764141583 [15279.289291] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15279.294236] LustreError: 510249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15279.706153] Lustre: server umount lustre-MDT0001 complete [15291.654944] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [15295.966727] Key type lgssc unregistered [15296.439155] LNet: 518103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15296.457850] LNetError: 518103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15296.483667] LNet: Removed LNI 192.168.202.117@tcp [15297.862057] Key type .llcrypt unregistered [15297.866931] Key type ._llcrypt unregistered [15323.469315] Key type ._llcrypt registered [15323.474720] Key type .llcrypt registered [15323.615402] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [15325.452536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15325.492307] alg: No test for adler32 (adler32-zlib) [15326.811878] Lustre: Lustre: Build Version: 2.17.57_114_gc92b566 [15327.216651] LNet: Added LNI 192.168.202.117@tcp [8/256/0/180] [15329.018719] Key type lgssc registered [15330.560311] Lustre: Echo OBD driver; http://www.lustre.org/ [15373.929075] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [15385.123621] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [15424.885434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15424.929427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15426.505801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15426.586269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15426.674663] Lustre: lustre-MDT0000: new disk, initializing [15426.943324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15426.985541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15433.826388] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [15446.822950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15447.019615] Lustre: 522570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15447.093485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15447.097703] Lustre: Skipped 1 previous similar message [15447.150382] Lustre: lustre-MDT0001: new disk, initializing [15447.239322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15447.281365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15447.287588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15450.957360] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [15461.200428] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15468.044779] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15476.049192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15476.392296] Lustre: lustre-OST0000: new disk, initializing [15476.396548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15476.402124] Lustre: 524540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15476.487287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15483.473803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15483.490487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15483.680110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15484.428124] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_default_debug -1 all [15496.777605] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15503.400958] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15503.955568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15509.283625] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15509.551426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15514.597237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15514.601827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15514.625634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15514.633933] Lustre: Skipped 1 previous similar message [15518.122617] Lustre: server umount lustre-OST0000 complete [15524.321750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15524.332395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15524.354860] Lustre: Skipped 1 previous similar message [15524.357727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15528.486204] Lustre: server umount lustre-MDT0000 complete [15532.311225] LustreError: 522561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857220 with bad export cookie 10088079134432721269 [15532.318854] LustreError: MGC192.168.202.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15532.323491] LustreError: 522561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15532.679400] Lustre: server umount lustre-MDT0001 complete [15560.941116] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing set_hostid [15570.859246] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing load_modules_local [15609.293269] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15356 sec ========= 04:48:15 (1788857295) [15611.088920] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:48:17 (1788857297) === [15614.750113] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:48:21 (1788857301) === [15662.720071] Lustre: DEBUG MARKER: oleg217-server.virtnet: executing unload_modules_local [15666.370679] Key type lgssc unregistered [15666.645512] LNet: 531186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15666.661264] LNetError: 531186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15666.688171] LNet: Removed LNI 192.168.202.117@tcp [15667.841424] Key type .llcrypt unregistered [15667.843072] Key type ._llcrypt unregistered