[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1128514079 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.003256] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008532] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011123] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.014069] *** VALIDATE selinux *** [ 0.023425] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028345] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031103] *** VALIDATE tmpfs *** [ 0.032439] *** VALIDATE proc *** [ 0.034261] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040027] Speculative Store Bypass: Vulnerable [ 0.043525] debug: unmapping init [mem 0xffffffff8e059000-0xffffffff8e060fff] [ 0.045243] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047299] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048127] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055261] rcu: Hierarchical SRCU implementation. [ 0.057512] smp: Bringing up secondary CPUs ... [ 0.058628] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.069426] smp: Brought up 1 node, 4 CPUs [ 0.071016] smpboot: Max logical packages: 1 [ 0.072012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.121327] node 0 deferred pages initialised in 47ms [ 0.125229] devtmpfs: initialized [ 0.126382] x86/mm: Memory block size: 128MB [ 0.131638] gcov: version magic: 0x41383552 [ 0.134385] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142356] pinctrl core: initialized pinctrl subsystem [ 0.144412] [ 0.144980] ************************************************************* [ 0.148015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151013] ** ** [ 0.154015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158018] ** ** [ 0.163014] ** This means that this kernel is built to expose internal ** [ 0.168014] ** IOMMU data structures, which may compromise security on ** [ 0.172010] ** your system. ** [ 0.174013] ** ** [ 0.177011] ** If you see this message and you are not debugging the ** [ 0.180014] ** kernel, report this immediately to your vendor! ** [ 0.183012] ** ** [ 0.186010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189068] ************************************************************* [ 0.193754] NET: Registered protocol family 16 [ 0.196447] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197000] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201101] cpuidle: using governor menu [ 0.202604] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204264] PCI: Using configuration type 1 for base access [ 0.206124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215081] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218497] cryptd: max_cpu_qlen set to 1000 [ 0.219320] ACPI: Added _OSI(Module Device) [ 0.220030] ACPI: Added _OSI(Processor Device) [ 0.221008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226811] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.231646] ACPI: Interpreter enabled [ 0.232046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233007] ACPI: Using IOAPIC for interrupt routing [ 0.234094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.236329] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246514] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.251055] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.255148] acpiphp: Slot [2] registered [ 0.256085] acpiphp: Slot [5] registered [ 0.257061] acpiphp: Slot [6] registered [ 0.258060] acpiphp: Slot [7] registered [ 0.259043] acpiphp: Slot [8] registered [ 0.260035] acpiphp: Slot [9] registered [ 0.261028] acpiphp: Slot [10] registered [ 0.261940] acpiphp: Slot [3] registered [ 0.262178] acpiphp: Slot [4] registered [ 0.262991] acpiphp: Slot [11] registered [ 0.264068] acpiphp: Slot [12] registered [ 0.264975] acpiphp: Slot [13] registered [ 0.266106] acpiphp: Slot [14] registered [ 0.266993] acpiphp: Slot [15] registered [ 0.268071] acpiphp: Slot [16] registered [ 0.269041] acpiphp: Slot [17] registered [ 0.269985] acpiphp: Slot [18] registered [ 0.271050] acpiphp: Slot [19] registered [ 0.272071] acpiphp: Slot [20] registered [ 0.273028] acpiphp: Slot [21] registered [ 0.274075] acpiphp: Slot [22] registered [ 0.275040] acpiphp: Slot [23] registered [ 0.276027] acpiphp: Slot [24] registered [ 0.277046] acpiphp: Slot [25] registered [ 0.277981] acpiphp: Slot [26] registered [ 0.279179] acpiphp: Slot [27] registered [ 0.280069] acpiphp: Slot [28] registered [ 0.281053] acpiphp: Slot [29] registered [ 0.281987] acpiphp: Slot [30] registered [ 0.283054] acpiphp: Slot [31] registered [ 0.283986] PCI host bridge to bus 0000:00 [ 0.285015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.290018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.292083] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.294069] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297205] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.299983] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.302985] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.310011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313038] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.316015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.318011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.319012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321390] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322512] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.324043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.326670] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.329826] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.339013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.343017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.346128] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.355012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.362013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.378012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.389229] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.396012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.408014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.433012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.443085] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.453022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.461015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.508038] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.523900] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.595029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.603059] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.617017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.628111] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.633013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.641021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.665014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.677405] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.684015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.691016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.708079] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.718107] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.720215] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.722457] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.724329] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.729243] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.735226] iommu: Default domain type: Passthrough [ 0.740043] SCSI subsystem initialized [ 0.741154] ACPI: bus type USB registered [ 0.743301] usbcore: registered new interface driver usbfs [ 0.746076] usbcore: registered new interface driver hub [ 0.749182] usbcore: registered new device driver usb [ 0.751209] pps_core: LinuxPPS API ver. 1 registered [ 0.752006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.756057] PTP clock support registered [ 0.758131] EDAC MC: Ver: 3.0.0 [ 0.759867] PCI: Using ACPI for IRQ routing [ 0.760853] NetLabel: Initializing [ 0.761008] NetLabel: domain hash size = 128 [ 0.762010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.763135] NetLabel: unlabeled traffic allowed by default [ 0.765551] vgaarb: loaded [ 0.767758] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.769011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.776291] clocksource: Switched to clocksource kvm-clock [ 0.899654] VFS: Disk quotas dquot_6.6.0 [ 0.900883] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.902917] *** VALIDATE ramfs *** [ 0.903958] *** VALIDATE hugetlbfs *** [ 0.905158] pnp: PnP ACPI init [ 0.907836] pnp: PnP ACPI: found 6 devices [ 0.923140] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.925769] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.927493] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.929190] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.930975] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.932932] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.935247] NET: Registered protocol family 2 [ 0.937188] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.941470] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.944533] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.948922] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.951726] TCP: Hash tables configured (established 65536 bind 65536) [ 0.953917] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.956383] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.958630] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960952] NET: Registered protocol family 1 [ 0.962884] RPC: Registered named UNIX socket transport module. [ 0.964431] RPC: Registered udp transport module. [ 0.965918] RPC: Registered tcp transport module. [ 0.967589] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.969747] NET: Registered protocol family 44 [ 0.971593] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.973721] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.975935] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.978254] PCI: CLS 0 bytes, default 64 [ 0.980459] Unpacking initramfs... [ 3.691661] debug: unmapping init [mem 0xffff95d4fcc54000-0xffff95d4fffbffff] [ 3.696088] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.698295] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.701292] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.302456] Initialise system trusted keyrings [ 4.303944] Key type blacklist registered [ 4.306586] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.316520] zbud: loaded [ 4.319528] *** VALIDATE nfs *** [ 4.320534] *** VALIDATE nfs4 *** [ 4.321788] pstore: using deflate compression [ 4.326114] Platform Keyring initialized [ 4.444934] NET: Registered protocol family 38 [ 4.446689] Key type asymmetric registered [ 4.447990] Asymmetric key parser 'x509' registered [ 4.449766] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.452893] io scheduler mq-deadline registered [ 4.454630] io scheduler kyber registered [ 4.455965] io scheduler bfq registered [ 4.457831] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.460490] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.463452] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.466901] ACPI: Power Button [PWRF] [ 4.472489] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.479396] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.494570] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.501087] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.827651] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.884295] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.912521] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.917972] Non-volatile memory driver v1.3 [ 4.919488] Linux agpgart interface v0.103 [ 4.947721] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 4.950477] vda: detected capacity change from 0 to 74645504 [ 4.969210] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.972075] vdb: detected capacity change from 0 to 1073741824 [ 4.989174] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.991933] vdc: detected capacity change from 0 to 2621440000 [ 5.021588] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.024548] vdd: detected capacity change from 0 to 2621440000 [ 5.053790] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.056671] vde: detected capacity change from 0 to 4294967296 [ 5.093389] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.100144] vdf: detected capacity change from 0 to 4294967296 [ 5.118200] libphy: Fixed MDIO Bus: probed [ 5.133103] usbcore: registered new interface driver usbserial_generic [ 5.135847] usbserial: USB Serial support registered for generic [ 5.138175] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.142988] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.145096] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.147673] mousedev: PS/2 mouse device common for all mice [ 5.151309] rtc_cmos 00:05: RTC can wake from S4 [ 5.154663] rtc_cmos 00:05: registered as rtc0 [ 5.156428] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.158810] intel_pstate: CPU model not supported [ 5.161327] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.164467] hid: raw HID events driver (C) Jiri Kosina [ 5.202974] usbcore: registered new interface driver usbhid [ 5.205082] usbhid: USB HID core driver [ 5.206723] drop_monitor: Initializing network drop monitor service [ 5.209157] Initializing XFRM netlink socket [ 5.211353] NET: Registered protocol family 10 [ 5.214193] Segment Routing with IPv6 [ 5.215555] NET: Registered protocol family 17 [ 5.217316] mpls_gso: MPLS GSO support [ 5.293989] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.301085] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.305932] RAS: Correctable Errors collector initialized. [ 5.308014] AVX version of gcm_enc/dec engaged. [ 5.309550] AES CTR mode by8 optimization enabled [ 5.386331] sched_clock: Marking stable (5386134792, 0)->(6557835276, -1171700484) [ 5.389849] registered taskstats version 1 [ 5.393473] Loading compiled-in X.509 certificates [ 5.395727] zswap: loaded using pool lzo/zbud [ 5.446331] Key type big_key registered [ 5.477318] Key type encrypted registered [ 5.480761] ima: No TPM chip found, activating TPM-bypass! [ 5.486529] ima: Allocated hash algorithm: sha1 [ 5.490446] ima: No architecture policies found [ 5.495217] evm: Initialising EVM extended attributes: [ 5.499678] evm: security.selinux [ 5.501680] evm: security.ima [ 5.504723] evm: security.capability [ 5.508721] evm: HMAC attrs: 0x1 [ 5.514186] rtc_cmos 00:05: setting system clock to 2026-07-16 18:01:03 UTC (1784224863) [ 5.531344] debug: unmapping init [mem 0xffffffff8f003000-0xffffffff8f1fffff] [ 5.581368] debug: unmapping init [mem 0xffffffff8dd82000-0xffffffff8e058fff] [ 5.588669] Write protecting the kernel read-only data: 28672k [ 5.592451] debug: unmapping init [mem 0xffffffff8c403000-0xffffffff8c5fffff] [ 5.595470] debug: unmapping init [mem 0xffffffff8cd14000-0xffffffff8cdfffff] [ 5.633365] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.642153] systemd[1]: Detected virtualization kvm. [ 5.644328] systemd[1]: Detected architecture x86-64. [ 5.646573] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.678350] systemd[1]: No hostname configured. [ 5.680098] systemd[1]: Set hostname to . [ 5.682440] random: systemd: uninitialized urandom read (16 bytes read) [ 5.684719] systemd[1]: Initializing machine ID from random generator. [ 5.961158] random: systemd: uninitialized urandom read (16 bytes read) [ 5.964486] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.972838] random: systemd: uninitialized urandom read (16 bytes read) [ 5.975632] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.984303] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.724660] device-mapper: uevent: version 1.0.3 [ 7.726538] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.735420] random: fast init done [ 10.036877] virtio_net virtio0 ens2: renamed from eth0 [ 10.351494] scsi host0: ata_piix [ 10.420437] scsi host1: ata_piix [ 10.422926] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.428731] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.021794] random: crng init done [ 15.023190] random: 7 urandom warning(s) missed due to ratelimiting [ 17.307836] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.261761] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.393473] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.938801] SELinux: Disabled at runtime. [ 23.046469] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.056377] systemd[1]: Detected virtualization kvm. [ 23.059864] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.550884] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.560765] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.581716] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.594646] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.612461] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.642928] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.649965] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 25.581164] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.710480] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.895936] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.016721] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.223533] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.331338] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit)[ 33.419623] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 34.182384] NFS: Registering the id_resolver key type [ 34.184731] Key type id_resolver registered [ 34.186728] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg216-server login: [ 52.535012] hrtimer: interrupt took 3008811 ns [ 79.290311] libcfs: loading out-of-tree module taints kernel. [ 79.365295] Key type ._llcrypt registered [ 79.366949] Key type .llcrypt registered [ 79.452511] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 92.595112] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 93.797963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.812352] alg: No test for adler32 (adler32-zlib) [ 95.015776] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 95.523555] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 97.199141] Key type lgssc registered [ 98.272269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 109.414341] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.332379] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 142.695351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.708623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.865936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.885129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.943544] Lustre: lustre-MDT0000: new disk, initializing [ 143.991162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.006917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.840449] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.176604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.251110] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 156.272786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 156.276490] Lustre: Skipped 1 previous similar message [ 156.334839] Lustre: lustre-MDT0001: new disk, initializing [ 156.367773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.383820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.391648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.176317] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.719764] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.601772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.747988] Lustre: lustre-OST0000: new disk, initializing [ 168.750923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.755743] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.798091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.650433] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.603562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.615719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.684394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 182.061960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.149458] Lustre: lustre-OST0001: new disk, initializing [ 182.152636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 182.158126] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.213154] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.097371] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.399887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 187.410606] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 187.448459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 194.723783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 201.060775] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 205.967155] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing check_logdir /tmp/testlogs/ [ 209.675239] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing yml_node [ 212.718758] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 214.434631] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 216.054491] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 217.033318] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 16 14:04:34 EDT 2026 [ 226.837326] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 227.758236] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 243.682322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 243.684303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 243.686771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 243.702986] Lustre: Skipped 2 previous similar messages [ 247.060086] Lustre: server umount lustre-MDT0000 complete [ 249.095151] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784225107 with bad export cookie 18392245579080309217 [ 249.099693] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 249.102719] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 253.920148] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 253.924394] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.931490] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 253.946703] Lustre: Skipped 1 previous similar message [ 253.951086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 253.956040] Lustre: Skipped 3 previous similar messages [ 254.945649] LustreError: 9221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 254.965651] LustreError: 9221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 255.415680] Lustre: server umount lustre-MDT0001 complete [ 257.875651] Lustre: server umount lustre-OST0000 complete [ 260.126599] Lustre: server umount lustre-OST0001 complete [ 263.916217] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 269.228489] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 288.433916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 288.564399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 288.584184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 288.645657] Lustre: lustre-MDT0000: new disk, initializing [ 288.694574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 288.703524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 290.772360] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 297.183633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 297.253182] Lustre: 15058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 297.298680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 297.301988] Lustre: Skipped 1 previous similar message [ 297.344331] Lustre: lustre-MDT0001: new disk, initializing [ 297.427470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 297.458609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 297.473492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 299.811911] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 305.832857] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 308.858809] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 312.592862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 312.751398] Lustre: 17026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 314.243244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 314.250749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 314.322446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 316.302727] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 322.575497] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 325.334344] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 325.493591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 328.180934] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 328.370700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 329.699991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.702106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 329.705959] Lustre: Skipped 1 previous similar message [ 329.707554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 329.713294] Lustre: Skipped 4 previous similar messages [ 334.817118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 334.821613] Lustre: Skipped 1 previous similar message [ 335.594857] Lustre: server umount lustre-OST0000 complete [ 338.400775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 338.409103] LustreError: Skipped 1 previous similar message [ 338.410242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.414948] Lustre: Skipped 1 previous similar message [ 339.936936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.939931] Lustre: Skipped 2 previous similar messages [ 343.588757] Lustre: server umount lustre-MDT0000 complete [ 345.056243] LustreError: 15860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.062889] LustreError: 15860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 345.261853] LustreError: 15051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784225203 with bad export cookie 18392245579080310750 [ 345.267151] LustreError: 15051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 345.267517] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.441239] Lustre: server umount lustre-MDT0001 complete [ 347.306185] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:06:45 (1784225205) [ 348.557300] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 349.429073] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:06:47 (1784225207) [ 352.670792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.917786] LustreError: 19435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.952697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.957348] Lustre: Skipped 1 previous similar message [ 354.798318] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 358.368531] LustreError: 19436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.461490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.816936] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 365.737561] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 368.882239] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 372.593871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 376.101192] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 380.310315] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 388.577128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 388.585946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.599813] Lustre: Skipped 1 previous similar message [ 388.606842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 390.461468] Lustre: server umount lustre-OST0000 complete [ 399.567959] Lustre: server umount lustre-MDT0000 complete [ 399.840094] LustreError: 19435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.855406] LustreError: 19435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 401.977111] LustreError: 19460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784225259 with bad export cookie 18392245579080311632 [ 401.979056] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 401.994142] LustreError: 19460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 402.259621] Lustre: server umount lustre-MDT0001 complete [ 409.363861] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 411.376925] Key type lgssc unregistered [ 411.587697] LNet: 22960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.593514] LNetError: 22960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 411.616663] LNet: Removed LNI 192.168.202.116@tcp [ 412.353290] Key type .llcrypt unregistered [ 412.355113] Key type ._llcrypt unregistered [ 428.227240] Key type ._llcrypt registered [ 428.228940] Key type .llcrypt registered [ 428.290125] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 428.996414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 429.024261] alg: No test for adler32 (adler32-zlib) [ 429.986297] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 430.158833] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 431.799201] Key type lgssc registered [ 432.535355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 438.181609] loop: module loaded [ 438.217440] loop0: detected capacity change from 0 to 204800000 [ 438.335966] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 438.343867] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 438.545233] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 442.335807] loop0: detected capacity change from 0 to 204800000 [ 442.451792] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 442.462043] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 442.470953] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 446.083104] loop0: detected capacity change from 0 to 204800000 [ 450.428049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 450.444345] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.611906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 451.664748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 451.739786] Lustre: lustre-MDT0000: new disk, initializing [ 451.807295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.820575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 455.089472] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 461.372202] loop1: detected capacity change from 0 to 204800000 [ 464.902902] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.140504] Lustre: lustre-OST07c6: new disk, initializing [ 465.142629] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 465.144598] Lustre: Skipped 1 previous similar message [ 465.147048] Lustre: 26121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.195407] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 465.720245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 465.728457] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 465.757956] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 469.148094] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 473.566140] Lustre: 25110:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 473.570839] Lustre: 25110:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 473.574649] Lustre: 25110:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 473.578786] Lustre: 25110:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 473.583518] Lustre: 25110:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 473.587875] Lustre: 25110:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 491.158512] Lustre: 25109:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 491.162623] Lustre: 25109:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 491.165220] Lustre: 25109:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 491.169493] Lustre: 25109:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 491.172969] Lustre: 25109:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 491.175881] Lustre: 25109:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 495.376548] Lustre: 25110:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 495.381605] Lustre: 25110:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 495.389787] Lustre: 25110:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 495.398491] Lustre: 25110:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 495.405518] Lustre: 25110:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 495.409798] Lustre: 25110:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 495.413661] Lustre: 25110:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 495.416682] Lustre: 25110:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 495.420611] Lustre: 25110:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 495.424935] Lustre: 25110:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 495.429120] Lustre: 25110:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 495.433112] Lustre: 25110:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 498.564540] Lustre: 26785:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 498.571486] Lustre: 26785:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 498.575533] Lustre: 26785:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 498.578931] Lustre: 26785:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 498.581869] Lustre: 26785:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 498.585413] Lustre: 26785:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 498.590477] Lustre: 26785:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 498.600306] Lustre: 26785:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 498.604586] Lustre: 26785:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 498.608157] Lustre: 26785:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 498.611145] Lustre: 26785:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 498.613388] Lustre: 26785:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 505.681052] Lustre: 25111:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 505.686428] Lustre: 25111:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 505.689804] Lustre: 25111:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 505.694628] Lustre: 25111:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 505.700493] Lustre: 25111:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 505.705144] Lustre: 25111:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 505.709424] Lustre: 25111:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 505.714268] Lustre: 25111:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 505.718699] Lustre: 25111:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 505.723315] Lustre: 25111:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 505.728395] Lustre: 25111:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 505.733032] Lustre: 25111:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 516.044974] Lustre: 25109:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 516.050171] Lustre: 25109:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 516.053535] Lustre: 25109:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 516.056952] Lustre: 25109:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 516.060419] Lustre: 25109:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 516.064345] Lustre: 25109:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 516.068099] Lustre: 25109:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 516.072804] Lustre: 25109:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 516.075788] Lustre: 25109:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 516.078770] Lustre: 25109:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 516.082429] Lustre: 25109:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 516.088696] Lustre: 25109:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 558.047931] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 558.050838] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.054732] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 563.168431] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 568.288787] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 572.384830] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 572.490725] Lustre: server umount lustre-OST07c6 complete [ 578.021885] Lustre: server umount lustre-MDT0000 complete [ 587.060835] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 588.604052] Key type lgssc unregistered [ 588.779620] LNet: 28675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.783112] LNetError: 28675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.790480] LNet: Removed LNI 192.168.202.116@tcp [ 589.287893] Key type .llcrypt unregistered [ 589.293721] Key type ._llcrypt unregistered [ 600.590585] Key type ._llcrypt registered [ 600.593801] Key type .llcrypt registered [ 600.664539] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 601.177645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.189963] alg: No test for adler32 (adler32-zlib) [ 602.128202] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 602.263608] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 603.855199] Key type lgssc registered [ 604.431651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 609.153202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 609.172603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.485433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.359515] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 618.543902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.803372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 621.308521] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 625.444057] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 628.482510] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 632.563432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 632.781179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.880867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 635.992308] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 640.202491] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 656.356048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 656.370166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.402227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 658.414582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 658.421419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 658.425814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.449257] Lustre: Skipped 1 previous similar message [ 661.594359] Lustre: server umount lustre-OST0000 complete [ 668.696919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 668.698485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 668.709746] Lustre: Skipped 1 previous similar message [ 668.721954] Lustre: Skipped 1 previous similar message [ 673.756898] Lustre: server umount lustre-MDT0000 complete [ 673.760433] LustreError: 29925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 673.782122] LustreError: 29925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 677.820726] LustreError: 29904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784225535 with bad export cookie 12763677154442659541 [ 677.823376] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.833661] LustreError: 29904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 678.294256] Lustre: server umount lustre-MDT0001 complete [ 693.569264] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 697.522144] Key type lgssc unregistered [ 697.998880] LNet: 33662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.017921] LNetError: 33662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.055366] LNet: Removed LNI 192.168.202.116@tcp [ 699.587466] Key type .llcrypt unregistered [ 699.589449] Key type ._llcrypt unregistered [ 730.637465] Key type ._llcrypt registered [ 730.639664] Key type .llcrypt registered [ 730.723413] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 733.022481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.047414] alg: No test for adler32 (adler32-zlib) [ 734.437765] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 734.943133] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 736.684576] Key type lgssc registered [ 738.446892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 747.748403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 747.762362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.205242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.664446] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 759.782282] LustreError: 34938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 759.797878] LustreError: 34938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 760.868605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.186226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 765.072535] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 772.257082] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 776.878625] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 782.807127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 783.221127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 788.260102] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 794.479364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 795.291929] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 804.837304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 804.845271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 804.855167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 804.871816] Lustre: Skipped 1 previous similar message [ 809.964114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 811.034975] Lustre: server umount lustre-OST0000 complete [ 815.075511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.080772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 815.090569] Lustre: Skipped 2 previous similar messages [ 815.104440] Lustre: Skipped 2 previous similar messages [ 820.193400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 820.209080] Lustre: Skipped 1 previous similar message [ 821.190824] Lustre: server umount lustre-MDT0000 complete [ 824.734145] LustreError: 34919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784225682 with bad export cookie 3286014123029034313 [ 824.737804] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 824.740374] LustreError: 34919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 825.115230] Lustre: server umount lustre-MDT0001 complete [ 834.702612] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 837.307360] Key type lgssc unregistered [ 837.655140] LNet: 38470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.666570] LNetError: 38470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.689268] LNet: Removed LNI 192.168.202.116@tcp [ 838.515375] Key type .llcrypt unregistered [ 838.517475] Key type ._llcrypt unregistered [ 858.483453] Key type ._llcrypt registered [ 858.485493] Key type .llcrypt registered [ 858.565549] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 859.564417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.691502] alg: No test for adler32 (adler32-zlib) [ 860.804823] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 861.115483] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 862.791360] Key type lgssc registered [ 863.775856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 870.603411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 870.626670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.018097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.718425] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 881.121939] LustreError: 39868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.130496] LustreError: 39868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 883.828965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.126136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 888.342773] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 894.613900] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 899.255724] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 905.259334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 905.497844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.110089] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 916.789426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 918.367946] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 925.157540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 925.165320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.177613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.534174] Lustre: server umount lustre-MDT0000 complete [ 927.201381] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.204744] LustreError: 39867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.224252] LustreError: 39867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 928.425821] LustreError: 40602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784225786 with bad export cookie 11720187712715890509 [ 928.433979] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 928.437362] LustreError: 40602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 930.370178] LustreError: 41962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.406311] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 930.410464] Lustre: Skipped 1 previous similar message [ 932.324081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.342734] Lustre: Skipped 1 previous similar message [ 932.358584] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 933.344160] LustreError: 39864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 933.355737] LustreError: 39864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 934.799906] Lustre: server umount lustre-MDT0001 complete [ 954.797501] Lustre: server umount lustre-OST0000 complete [ 968.112602] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 970.272557] Key type lgssc unregistered [ 970.546844] LNet: 43727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.553184] LNetError: 43727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.566226] LNet: Removed LNI 192.168.202.116@tcp [ 971.358193] Key type .llcrypt unregistered [ 971.363483] Key type ._llcrypt unregistered [ 990.100612] Key type ._llcrypt registered [ 990.102266] Key type .llcrypt registered [ 990.195994] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 991.405063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 991.441460] alg: No test for adler32 (adler32-zlib) [ 992.431138] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 992.622527] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 994.336626] Key type lgssc registered [ 995.378175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1002.758716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1002.788533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.347129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1009.014184] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1014.756466] LustreError: 45121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.786608] LustreError: 45121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1018.056866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.723757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1023.090840] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1030.616697] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1036.145802] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1042.063498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1042.333939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.003379] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1049.523395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1055.315344] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1063.014559] Lustre: lustre-OST0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 1063.912610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1063.930574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.942872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1064.933776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1064.945288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.962742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1064.974294] Lustre: Skipped 1 previous similar message [ 1067.992168] Lustre: server umount lustre-OST0000 complete [ 1085.408816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.426255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1085.436189] Lustre: Skipped 1 previous similar message [ 1090.528832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1090.545147] Lustre: Skipped 1 previous similar message [ 1090.859303] Lustre: server umount lustre-MDT0000 complete [ 1094.614748] LustreError: 45103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784225952 with bad export cookie 4416815515188423900 [ 1094.621613] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.625410] LustreError: 45103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1101.014301] Lustre: server umount lustre-MDT0001 complete [ 1110.306795] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1112.673394] Key type lgssc unregistered [ 1112.910528] LNet: 48830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.923490] LNetError: 48830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.946944] LNet: Removed LNI 192.168.202.116@tcp [ 1113.649495] Key type .llcrypt unregistered [ 1113.651346] Key type ._llcrypt unregistered [ 1133.235278] Key type ._llcrypt registered [ 1133.238050] Key type .llcrypt registered [ 1133.329648] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1134.064383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1134.099834] alg: No test for adler32 (adler32-zlib) [ 1135.116157] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 1135.310486] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1136.967197] Key type lgssc registered [ 1137.866202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1144.692862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1144.719029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1146.048903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.213112] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1155.652157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.914107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1159.627944] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1166.233814] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1171.633582] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1178.072724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1178.347190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.129981] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1191.235330] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1191.685282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1197.419946] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1200.855691] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1213.492264] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-1614h21m09s [ 1215.129864] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1217.504085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1217.508215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.544480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1217.553688] Lustre: Skipped 2 previous similar messages [ 1218.595793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1218.602865] Lustre: Skipped 2 previous similar messages [ 1219.665057] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 1223.866945] Lustre: server umount lustre-MDT0000 complete [ 1224.680511] LustreError: 50226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1224.696673] LustreError: 50226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1225.786631] LustreError: 50220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.748651] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1229.808639] LustreError: 50220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.834453] LustreError: 50220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1231.934414] LustreError: 50220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.968039] LustreError: 50220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1235.241977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.342926] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.563265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.646883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1236.608655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1239.970781] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1243.150082] Lustre: DEBUG MARKER: Wait for df (36622) ... [ 1244.849431] Lustre: DEBUG MARKER: done [ 1250.789407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1250.797618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.806913] Lustre: Skipped 2 previous similar messages [ 1250.817780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1250.826858] Lustre: Skipped 1 previous similar message [ 1252.328231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.345435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1257.444041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1257.448978] Lustre: Skipped 2 previous similar messages [ 1263.071297] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1263.222878] Lustre: server umount lustre-OST0000 complete [ 1267.681609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1267.681671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.694087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.703220] Lustre: Skipped 1 previous similar message [ 1267.716807] Lustre: Skipped 3 previous similar messages [ 1272.680668] Lustre: server umount lustre-MDT0000 complete [ 1273.829212] LustreError: 50220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.840725] LustreError: 50220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1275.689765] LustreError: 52852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784226133 with bad export cookie 10416604823778262492 [ 1275.694442] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.699851] LustreError: 52852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1275.965429] Lustre: server umount lustre-MDT0001 complete [ 1284.565899] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1286.872827] Key type lgssc unregistered [ 1287.154548] LNet: 55150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.158333] LNetError: 55150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.246085] LNet: Removed LNI 192.168.202.116@tcp [ 1289.087724] Key type .llcrypt unregistered [ 1289.089758] Key type ._llcrypt unregistered [ 1319.903116] Key type ._llcrypt registered [ 1319.904978] Key type .llcrypt registered [ 1319.972904] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1320.741107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.761570] alg: No test for adler32 (adler32-zlib) [ 1321.719396] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 1321.926928] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1323.567164] Key type lgssc registered [ 1324.374702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1330.458511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1330.480406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.696561] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1331.715223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1331.832885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.783183] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1340.635100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.727576] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1340.754345] Lustre: 57341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1340.971151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1343.742988] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1348.938677] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1353.204083] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1357.880856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1357.972237] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1358.074716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.469899] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1366.138265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1368.237489] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1373.446163] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-1614h23m50s [ 1374.542185] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1380.187201] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1382.139848] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1382.184541] LustreError: 57349:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1402.472628] Lustre: lustre-MDT0000: Client f314df71-b410-4d1d-aa08-09656000f15c (at 192.168.202.16@tcp) reconnecting [ 1422.271116] LustreError: 57349:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1424.050949] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1425.917692] Lustre: DEBUG MARKER: done [ 1432.546214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1432.550133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.558886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1433.059219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1433.059368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.067782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1436.255754] Lustre: server umount lustre-OST0000 complete [ 1443.296778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1443.301414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.302923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.308218] Lustre: Skipped 1 previous similar message [ 1443.318406] Lustre: Skipped 2 previous similar messages [ 1450.464594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.469088] Lustre: Skipped 2 previous similar messages [ 1453.023287] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1453.160478] Lustre: server umount lustre-MDT0000 complete [ 1456.042345] LustreError: 57332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784226314 with bad export cookie 3314300271034535839 [ 1456.048898] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.064035] LustreError: 57332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1456.385922] Lustre: server umount lustre-MDT0001 complete [ 1464.568117] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1466.564843] Key type lgssc unregistered [ 1466.772821] LNet: 61665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.776355] LNetError: 61665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.783995] LNet: Removed LNI 192.168.202.116@tcp [ 1467.398794] Key type .llcrypt unregistered [ 1467.401269] Key type ._llcrypt unregistered [ 1498.031380] Key type ._llcrypt registered [ 1498.034468] Key type .llcrypt registered [ 1498.120433] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1498.730416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1498.768781] alg: No test for adler32 (adler32-zlib) [ 1499.751924] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 1499.902598] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1501.527199] Key type lgssc registered [ 1502.255636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1504.442769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1504.453398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.631815] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1505.652049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1505.752222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1506.364810] Lustre: Failing over lustre-MDT0000 [ 1506.524489] Lustre: server umount lustre-MDT0000 complete [ 1510.607220] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:26:08 (1784226368) [ 1514.465384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.853690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.516612] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1523.284857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.362155] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1523.381819] Lustre: 64441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1523.433339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1523.548080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1526.104176] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1530.618459] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1534.469170] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1538.942969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1539.065343] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1539.204640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.494690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1543.245281] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1547.764842] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1552.830114] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1557.992581] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.998919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1558.005747] Lustre: Skipped 2 previous similar messages [ 1558.018539] Lustre: Skipped 2 previous similar messages [ 1563.069497] Lustre: server umount lustre-MDT0000 complete [ 1565.726383] LustreError: 66301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784226423 with bad export cookie 5098261320374109671 [ 1565.730795] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1565.738818] LustreError: 66301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1565.941887] Lustre: server umount lustre-MDT0001 complete [ 1568.427983] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1571.482328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1577.272196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.607225] Lustre: 63368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784226426/real 1784226426] req@0000000082215653 x1870896936552320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784226442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1584.622115] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.655464] Lustre: 63368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784226428/real 1784226428] req@00000000ae5a5b48 x1870896936552448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784226444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1589.727195] Lustre: 63366:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784226431/real 1784226431] req@000000005b410745 x1870896936552576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784226447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1590.751713] LustreError: 63364:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000065598c4a x1870896936553344/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1591.010521] LustreError: 66285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1591.023934] LustreError: 66285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1591.089596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.164639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1591.775105] Lustre: 63365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784226433/real 1784226433] req@00000000c32f40a2 x1870896936552832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784226449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1593.121053] LustreError: 66286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.145685] LustreError: 68205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.153354] LustreError: 68205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1594.183276] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1596.193304] Lustre: 63368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784226438/real 1784226438] req@000000001d59a049 x1870896936552960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784226454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1599.456113] LustreError: 68206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1599.467881] LustreError: 68206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1600.457285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.779505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1604.014044] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1606.120265] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1608.656814] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1611.780403] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1613.293055] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1616.353602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1616.360756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.370840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.402218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.404192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.526568] Lustre: server umount lustre-MDT0000 complete [ 1623.521087] LustreError: 68205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1623.532210] LustreError: 68205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1623.786593] LustreError: 64434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784226481 with bad export cookie 5098261320374111946 [ 1623.789259] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.791618] LustreError: 64434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1623.965775] Lustre: server umount lustre-MDT0001 complete [ 1627.798236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1633.389900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.965259] LustreError: 66286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.027115] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.045913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.046258] Lustre: Skipped 1 previous similar message [ 1634.094683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1636.537170] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1642.316477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.502348] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1645.535085] Lustre: 63368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784226486/real 1784226486] req@00000000c0f17066 x1870896936577024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784226502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1647.595618] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1650.753932] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1654.889806] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1661.556942] Lustre: server umount lustre-MDT0000 complete [ 1662.952678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1662.958887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.968570] LustreError: 66286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.979318] LustreError: 66286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1663.783937] LustreError: 66301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784226521 with bad export cookie 5098261320374113325 [ 1663.790156] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1663.797030] LustreError: 66301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1663.969681] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1663.973174] Lustre: Skipped 2 previous similar messages [ 1666.091989] Lustre: server umount lustre-MDT0001 complete [ 1670.169711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1672.278766] Lustre: DEBUG MARKER: files compared the same [ 1673.402577] Lustre: server umount lustre-OST0000 complete [ 1682.678890] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1684.178846] Key type lgssc unregistered [ 1684.371582] LNet: 74154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.374631] LNetError: 74154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.413360] LNet: Removed LNI 192.168.202.116@tcp [ 1685.811187] Key type .llcrypt unregistered [ 1685.812247] Key type ._llcrypt unregistered [ 1699.574875] Key type ._llcrypt registered [ 1699.576749] Key type .llcrypt registered [ 1699.626793] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1700.083313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1700.090615] alg: No test for adler32 (adler32-zlib) [ 1701.001416] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 1701.113330] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1702.711414] Key type lgssc registered [ 1703.292764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1707.477605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1707.485141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.705575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.045238] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug +malloc all [ 1716.566476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.752363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1719.373677] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug +malloc all [ 1724.075073] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1727.603760] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1731.735661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1731.937130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.995426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1735.316407] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug +malloc all [ 1738.849853] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1749.472710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1749.476764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1749.483946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1751.521064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1751.522112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.522398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1753.211406] Lustre: server umount lustre-OST0000 complete [ 1756.641088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.644166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1756.648348] Lustre: Skipped 1 previous similar message [ 1756.651870] Lustre: Skipped 1 previous similar message [ 1761.650151] Lustre: server umount lustre-MDT0000 complete [ 1761.770721] LustreError: 75550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.781165] LustreError: 75550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1763.948619] LustreError: 75532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784226621 with bad export cookie 16350057204629629051 [ 1763.950619] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.957233] LustreError: 75532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1764.093526] Lustre: server umount lustre-MDT0001 complete [ 1771.185747] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1772.730594] Key type lgssc unregistered [ 1772.892300] LNet: 79078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.898652] LNetError: 79078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.910807] LNet: Removed LNI 192.168.202.116@tcp [ 1773.394511] Key type .llcrypt unregistered [ 1773.396861] Key type ._llcrypt unregistered [ 1787.615703] Key type ._llcrypt registered [ 1787.618098] Key type .llcrypt registered [ 1787.697801] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1788.258478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1788.382609] alg: No test for adler32 (adler32-zlib) [ 1789.316651] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 1789.511865] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1791.143825] Key type lgssc registered [ 1791.780458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.036534] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1796.054214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1811.551389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1816.671465] LustreError: 80448:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 1843.295263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.099118] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1849.843348] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1853.926504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.013399] Lustre: *** cfs_fail_loc=706, val=0*** [ 1854.298472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.335434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1856.625177] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1861.402375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.634555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1864.015518] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1868.368926] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1871.536430] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1874.915637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1874.920656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.927685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1876.963877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.963945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1876.973301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1876.976828] Lustre: Skipped 1 previous similar message [ 1879.164058] Lustre: server umount lustre-OST0000 complete [ 1882.085437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.088370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1882.102321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.200449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.203698] Lustre: Skipped 2 previous similar messages [ 1887.652647] Lustre: server umount lustre-MDT0000 complete [ 1889.839303] LustreError: 80456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784226747 with bad export cookie 8959802276684596520 [ 1889.842580] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1889.843561] LustreError: 80456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1890.011206] Lustre: server umount lustre-MDT0001 complete [ 1895.880487] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1897.372752] Key type lgssc unregistered [ 1897.544668] LNet: 84059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.549299] LNetError: 84059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.558638] LNet: Removed LNI 192.168.202.116@tcp [ 1898.019674] Key type .llcrypt unregistered [ 1898.021295] Key type ._llcrypt unregistered [ 1911.385712] Key type ._llcrypt registered [ 1911.387347] Key type .llcrypt registered [ 1911.443319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1911.922497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1911.997351] alg: No test for adler32 (adler32-zlib) [ 1912.909017] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 1913.041910] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1914.655207] Key type lgssc registered [ 1915.190562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1918.738524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1918.750094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.622704] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1925.189506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.484862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1927.816919] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1930.722311] LustreError: 86139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.731530] LustreError: 86139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1932.549914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1932.731694] LustreError: 86137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.739355] LustreError: 86137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1932.753535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.594370] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1935.841784] LustreError: 86143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.858687] LustreError: 86143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1940.034314] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1940.040768] Lustre: MGS: Client 5d06c76d-9e7c-4580-ab77-c2f745a7dd88 (at 0@lo) reconnecting [ 1940.041806] LustreError: 86142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.043594] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7c1103d852082dfd [ 1940.048110] LustreError: 86142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1940.050969] Lustre: MGC192.168.202.116@tcp: Connection restored to 0@lo (at 0@lo) [ 1940.065169] LustreError: 86153:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1940.186465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.231617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1942.039267] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1955.807996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1955.811701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.815665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1955.820544] Lustre: Skipped 1 previous similar message [ 1955.828093] Lustre: Skipped 1 previous similar message [ 1957.348831] Lustre: server umount lustre-OST0000 complete [ 1960.928502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1960.933098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.937777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.938277] Lustre: Skipped 1 previous similar message [ 1960.942369] Lustre: Skipped 1 previous similar message [ 1965.319454] Lustre: server umount lustre-MDT0000 complete [ 1966.049451] LustreError: 86143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.058739] LustreError: 86143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1967.453145] Lustre: server umount lustre-MDT0001 complete [ 1969.412323] Lustre: server umount MGS complete [ 1975.657307] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:33:53 (1784226833) [ 1990.335761] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 1994.403223] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2011.804654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.879702] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2019.861708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.040527] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2020.075669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2020.125712] Lustre: lustre-MDT0001: new disk, initializing [ 2020.171395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.191701] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2020.198463] Lustre: 93303:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2020.206427] LustreError: 93303:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2022.352027] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2027.918151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.055175] Lustre: lustre-OST0000: new disk, initializing [ 2028.057625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2028.061544] Lustre: 94068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2028.100465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.048971] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2036.118563] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2038.306113] Lustre: 94884:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000994e1db9 x1870897370663680/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784226912 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2039.360628] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.366134] Lustre: MGS: Client 522a29a0-9455-4868-ab2b-23234482b068 (at 0@lo) reconnecting [ 2039.397546] Lustre: 92571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2039.408263] LustreError: 93289:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2039.412618] LustreError: 93289:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2039.426697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2039.475154] Lustre: lustre-MDT0000: new disk, initializing [ 2039.545231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.559098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2040.609410] Lustre: 95138:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000042a70d49 x1870897370669184/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1784226914 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2040.676220] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2041.544806] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2047.398089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2047.402724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2049.702321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2049.706661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2049.767487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2073.060877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2073.064972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.071773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2075.616843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2075.620318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.627790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2077.677051] Lustre: server umount lustre-OST0000 complete [ 2081.760046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2081.763879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.767555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.768751] Lustre: Skipped 1 previous similar message [ 2081.774325] Lustre: Skipped 2 previous similar messages [ 2085.667721] Lustre: server umount lustre-MDT0000 complete [ 2086.884521] LustreError: 95548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.904326] LustreError: 95548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2102.239239] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2102.352959] Lustre: server umount lustre-MDT0001 complete [ 2104.336565] Lustre: server umount MGS complete [ 2110.512087] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:36:08 (1784226968) [ 2119.566456] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2121.019079] Key type lgssc unregistered [ 2121.198442] LNet: 97634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.202829] LNetError: 97634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.215619] LNet: Removed LNI 192.168.202.116@tcp [ 2121.641194] Key type .llcrypt unregistered [ 2121.643071] Key type ._llcrypt unregistered [ 2129.993903] Key type ._llcrypt registered [ 2129.995483] Key type .llcrypt registered [ 2130.046877] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2130.418394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2130.441718] alg: No test for adler32 (adler32-zlib) [ 2131.328632] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 2131.438358] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2133.031863] Key type lgssc registered [ 2133.567832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2137.037820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2137.052426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.250155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.846675] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2143.318219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.485381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2145.249466] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2148.496107] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2150.978713] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2154.046892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2154.254405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.967329] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2160.574073] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2163.494941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2179.041937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.045355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.059196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.063175] Lustre: Skipped 1 previous similar message [ 2184.160202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.164769] Lustre: Skipped 1 previous similar message [ 2188.255688] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2188.329595] Lustre: server umount lustre-OST0000 complete [ 2194.400546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.402268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.407881] Lustre: Skipped 2 previous similar messages [ 2196.443846] Lustre: server umount lustre-MDT0000 complete [ 2198.285258] LustreError: 98813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227056 with bad export cookie 1156177324885656709 [ 2198.289718] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2198.293470] LustreError: 98813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2198.499358] Lustre: server umount lustre-MDT0001 complete [ 2204.463404] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2206.147523] Key type lgssc unregistered [ 2206.344633] LNet: 102386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.349707] LNetError: 102386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.368720] LNet: Removed LNI 192.168.202.116@tcp [ 2206.844670] Key type .llcrypt unregistered [ 2206.847584] Key type ._llcrypt unregistered [ 2215.187936] Key type ._llcrypt registered [ 2215.189560] Key type .llcrypt registered [ 2215.248659] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2215.938384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2215.945694] alg: No test for adler32 (adler32-zlib) [ 2216.849088] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 2216.960106] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2218.567303] Key type lgssc registered [ 2219.080268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2221.763901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2221.766052] LustreError: 103448:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2221.766103] Lustre: Skipped 1 previous similar message [ 2221.769511] LustreError: 103447:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2221.771729] LustreError: 103448:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2221.773885] LustreError: 103448:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2221.778288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.778417] LustreError: 103448:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2222.969577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.176325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.277894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2227.801796] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2229.580498] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing lsmod [ 2230.616805] LustreError: 104651:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2230.617360] LustreError: 104652:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2230.620379] LustreError: 104651:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 2230.622798] LustreError: 104651:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2230.625323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2230.626147] LustreError: 104651:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2230.777042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.758055] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.759551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.761225] Lustre: Skipped 2 previous similar messages [ 2231.762822] Lustre: Skipped 1 previous similar message [ 2236.897858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.899641] Lustre: Skipped 3 previous similar messages [ 2242.017238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.019209] Lustre: Skipped 2 previous similar messages [ 2246.111221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2246.175537] Lustre: server umount lustre-MDT0000 complete [ 2247.137691] LustreError: 103485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.141964] LustreError: 103485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2247.460154] LustreError: 104692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227105 with bad export cookie 12815528380366370091 [ 2247.462147] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.463309] LustreError: 104692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2249.183342] LustreError: 103914:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b99f11dc x1870897688364800/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2249.234787] Lustre: server umount lustre-MDT0001 complete [ 2260.531933] Lustre: server umount lustre-OST0000 complete [ 2264.162757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.411384] LustreError: 105716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.446487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.155079] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2269.666163] LustreError: 105717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.896658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.091896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2271.957700] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2274.999778] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2277.318370] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2280.456408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2280.652674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.241176] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2286.364908] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2289.898593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2300.383979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2300.388464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.394831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2301.409225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.773541] Lustre: server umount lustre-OST0000 complete [ 2311.136038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2311.139035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.145140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.147248] Lustre: Skipped 2 previous similar messages [ 2313.773888] Lustre: server umount lustre-MDT0000 complete [ 2315.605228] LustreError: 105698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227173 with bad export cookie 12815528380366370777 [ 2315.609026] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.609836] LustreError: 105698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2315.741385] Lustre: server umount lustre-MDT0001 complete [ 2320.721935] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2322.253751] Key type lgssc unregistered [ 2322.426841] LNet: 109266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.433204] LNetError: 109266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2322.442164] LNet: Removed LNI 192.168.202.116@tcp [ 2322.886252] Key type .llcrypt unregistered [ 2322.887627] Key type ._llcrypt unregistered [ 2335.145186] Key type ._llcrypt registered [ 2335.147125] Key type .llcrypt registered [ 2335.214265] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2335.841831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.854862] alg: No test for adler32 (adler32-zlib) [ 2336.754232] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 2336.860710] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2338.479159] Key type lgssc registered [ 2339.113432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2343.050419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2343.064595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.281465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.329836] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2350.631493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.806485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2352.799837] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2356.259487] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2358.651049] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2361.282652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2361.444151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.906736] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2366.998992] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2369.644160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2379.529507] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2380.473543] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2385.375919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2385.379800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.387213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.401190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2386.404141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.405071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.750611] Lustre: server umount lustre-OST0000 complete [ 2391.521081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.521831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2391.526182] Lustre: Skipped 1 previous similar message [ 2391.528511] Lustre: Skipped 1 previous similar message [ 2395.442366] Lustre: server umount lustre-MDT0000 complete [ 2396.640078] LustreError: 110660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.646112] LustreError: 110660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2397.165244] LustreError: 113210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227255 with bad export cookie 16047667248083515944 [ 2397.167771] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.170461] LustreError: 113210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2397.312494] Lustre: server umount lustre-MDT0001 complete [ 2401.704234] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2402.909091] Key type lgssc unregistered [ 2403.081604] LNet: 114315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.085514] LNetError: 114315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2404.136332] LNet: Removed LNI 192.168.202.116@tcp [ 2404.521239] Key type .llcrypt unregistered [ 2404.522948] Key type ._llcrypt unregistered [ 2413.396381] Key type ._llcrypt registered [ 2413.397891] Key type .llcrypt registered [ 2413.445496] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2413.916740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2413.929857] alg: No test for adler32 (adler32-zlib) [ 2414.796833] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 2414.911840] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2416.519161] Key type lgssc registered [ 2416.992886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2420.487253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2420.498472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.748243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.684477] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2427.327633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.497151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2429.112270] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2432.123591] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2434.185419] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2436.888417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2437.058426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.368579] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2442.258631] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2446.315504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2451.426875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.428622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2451.431122] Lustre: Skipped 1 previous similar message [ 2451.431536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2451.434038] Lustre: Skipped 1 previous similar message [ 2452.841136] Lustre: server umount lustre-OST0000 complete [ 2456.542380] Lustre: server umount lustre-MDT0000 complete [ 2456.545187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.550363] LustreError: 115509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.557498] LustreError: 115509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2458.024229] LustreError: 115495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227316 with bad export cookie 3987947283001059786 [ 2458.026834] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.029457] LustreError: 115495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2458.160723] Lustre: server umount lustre-MDT0001 complete [ 2462.478277] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2463.750032] Key type lgssc unregistered [ 2463.907658] LNet: 119041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.911252] LNetError: 119041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2463.923524] LNet: Removed LNI 192.168.202.116@tcp [ 2464.312905] Key type .llcrypt unregistered [ 2464.314758] Key type ._llcrypt unregistered [ 2475.254255] Key type ._llcrypt registered [ 2475.255856] Key type .llcrypt registered [ 2475.315076] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2475.811368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2475.822894] alg: No test for adler32 (adler32-zlib) [ 2476.693228] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 2476.809240] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2478.399226] Key type lgssc registered [ 2478.812263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2481.792759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2481.802759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.973779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.478927] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2487.737784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.874352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2489.348513] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2491.870196] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2493.740758] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2496.159954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2496.295747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.565646] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2501.372078] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2508.581418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2514.741496] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2516.102655] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2516.105374] Lustre: Skipped 2 previous similar messages [ 2517.725108] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2517.728294] Lustre: Skipped 2 previous similar messages [ 2524.682982] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2524.695192] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2527.334746] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.16@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2527.339614] Lustre: Skipped 2 previous similar messages [ 2529.797353] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.16@tcp 192.168.0.[2,10]@tcp [ 2536.096288] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.16/24@tcp in log lustre-MDT0000 [ 2536.099657] Lustre: Skipped 2 previous similar messages [ 2541.570396] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.16/24@tcp [ 2541.572652] Lustre: Skipped 1 previous similar message [ 2547.603996] Lustre: server umount lustre-OST0000 complete [ 2549.216259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2549.219199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.224472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2549.728821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.735402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.848251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.851418] Lustre: Skipped 1 previous similar message [ 2558.968585] Lustre: server umount lustre-MDT0000 complete [ 2559.969232] LustreError: 122919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.975822] LustreError: 122919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2560.326145] LustreError: 123081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227418 with bad export cookie 3552162052413975824 [ 2560.329505] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.331037] LustreError: 123081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2570.774281] Lustre: server umount lustre-MDT0001 complete [ 2574.853391] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2576.040422] Key type lgssc unregistered [ 2576.175363] LNet: 125529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.178959] LNetError: 125529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2576.190027] LNet: Removed LNI 192.168.202.116@tcp [ 2576.553927] Key type .llcrypt unregistered [ 2576.555634] Key type ._llcrypt unregistered [ 2587.644948] Key type ._llcrypt registered [ 2587.646022] Key type .llcrypt registered [ 2587.687382] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2588.202681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2588.220306] alg: No test for adler32 (adler32-zlib) [ 2589.097502] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 2589.201234] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2590.783160] Key type lgssc registered [ 2591.195362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2594.323522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2594.333168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.525065] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2595.528735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.537627] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.16/24@tcp [ 2597.193127] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2600.727190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.881267] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2600.893098] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.16/24@tcp [ 2600.922593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2602.735511] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2606.471364] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2609.219214] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2612.367995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2612.511425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.043712] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2618.301169] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2620.714768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2631.137900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2631.138093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.140757] LustreError: Skipped 1 previous similar message [ 2631.145284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2631.147972] Lustre: Skipped 1 previous similar message [ 2635.674354] Lustre: server umount lustre-OST0000 complete [ 2641.377089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.378767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2641.382020] Lustre: Skipped 2 previous similar messages [ 2643.486487] Lustre: server umount lustre-MDT0000 complete [ 2645.104118] LustreError: 127047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227503 with bad export cookie 14423690081589020311 [ 2645.106842] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2645.108260] LustreError: 127047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2645.250806] Lustre: server umount lustre-MDT0001 complete [ 2649.462107] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2650.625821] Key type lgssc unregistered [ 2650.761517] LNet: 130647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2650.764544] LNetError: 130647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2650.772448] LNet: Removed LNI 192.168.202.116@tcp [ 2651.136505] Key type .llcrypt unregistered [ 2651.138398] Key type ._llcrypt unregistered [ 2661.565196] Key type ._llcrypt registered [ 2661.566197] Key type .llcrypt registered [ 2661.607398] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2668.519941] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2668.879338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2668.888569] alg: No test for adler32 (adler32-zlib) [ 2669.781312] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 2669.877117] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2671.471166] Key type lgssc registered [ 2672.153572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2687.780887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2687.789341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.923662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2688.938078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2688.974265] Lustre: lustre-MDT0000: new disk, initializing [ 2689.006273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.015299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2690.827705] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2696.150334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2696.208205] Lustre: 134037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2696.239819] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2696.242805] Lustre: Skipped 1 previous similar message [ 2696.285375] Lustre: lustre-MDT0001: new disk, initializing [ 2696.331369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2696.348320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2696.352261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2698.060482] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2702.590634] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2705.017764] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2708.236626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.389526] Lustre: lustre-OST0000: new disk, initializing [ 2708.391733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2708.394698] Lustre: 136005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2708.433673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.127261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2709.131320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2709.180239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2710.777723] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2714.991761] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2716.516309] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2716.597988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2718.090711] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2718.169842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2719.713813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.714140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2719.718360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.720069] LustreError: Skipped 1 previous similar message [ 2719.723074] Lustre: Skipped 1 previous similar message [ 2724.833853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2724.836138] Lustre: Skipped 1 previous similar message [ 2724.954080] Lustre: server umount lustre-OST0000 complete [ 2727.392205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2727.395580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.400947] Lustre: Skipped 1 previous similar message [ 2727.403409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.952442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.954474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.389422] Lustre: server umount lustre-MDT0000 complete [ 2733.918454] LustreError: 134029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227591 with bad export cookie 422153844693569 [ 2733.919552] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2733.922779] LustreError: 134029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2734.045531] Lustre: server umount lustre-MDT0001 complete [ 2737.127740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.318248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.898641] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2742.253377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.377768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2743.944138] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2746.489893] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2748.427179] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2750.976981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2751.110973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.343058] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2756.200766] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2757.878732] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2763.083565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2764.753530] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2764.865427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2769.576469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2769.655363] Lustre: lustre-OST0001: new disk, initializing [ 2769.658153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2769.661722] Lustre: 141624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2769.702684] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2771.931829] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2775.708706] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2779.664290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2779.668797] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2779.719489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2779.887137] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2781.552914] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2781.649942] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2783.647171] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8af2c2c90800.ost_server_uuid 50 [ 2784.150760] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8af2c2c90800.ost_server_uuid in FULL state after 0 sec [ 2789.860359] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.862232] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2789.864659] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2789.864664] Lustre: Skipped 1 previous similar message [ 2789.865794] Lustre: Skipped 1 previous similar message [ 2792.257575] Lustre: server umount lustre-OST0001 complete [ 2794.977059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.979669] LustreError: 140039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2794.982715] Lustre: Skipped 1 previous similar message [ 2794.988621] LustreError: 140039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2799.569285] Lustre: server umount lustre-OST0000 complete [ 2803.679926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2803.682832] LustreError: Skipped 1 previous similar message [ 2803.685035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.691603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2803.693827] Lustre: Skipped 3 previous similar messages [ 2807.065166] Lustre: server umount lustre-MDT0000 complete [ 2808.340349] LustreError: 138203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227666 with bad export cookie 422153844694451 [ 2808.344456] LustreError: 138203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2808.344938] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2808.473398] Lustre: server umount lustre-MDT0001 complete [ 2815.272327] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2816.381721] Key type lgssc unregistered [ 2816.504381] LNet: 144649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2816.507331] LNetError: 144649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2816.518282] LNet: Removed LNI 192.168.202.116@tcp [ 2816.835400] Key type .llcrypt unregistered [ 2816.836440] Key type ._llcrypt unregistered [ 2831.362660] Key type ._llcrypt registered [ 2831.364201] Key type .llcrypt registered [ 2831.409157] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2837.637424] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2838.039334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2838.097540] alg: No test for adler32 (adler32-zlib) [ 2838.965074] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 2839.064918] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2840.647188] Key type lgssc registered [ 2841.074321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2853.520476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2853.527607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.628356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2854.642577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2854.679814] Lustre: lustre-MDT0000: new disk, initializing [ 2854.709185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.717157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2856.177790] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2860.525024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.568448] Lustre: 148773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2860.594898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2860.597649] Lustre: Skipped 1 previous similar message [ 2860.640173] Lustre: lustre-MDT0001: new disk, initializing [ 2860.677285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2860.691186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2860.694516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2862.138356] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2866.298231] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2868.508963] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2871.099384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2871.202520] Lustre: lustre-OST0000: new disk, initializing [ 2871.204700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2871.208069] Lustre: 150741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2871.241380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.710784] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2877.891870] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2880.507626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2880.510964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2880.547424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2885.059834] Lustre: Failing over lustre-OST0000 [ 2885.604117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2885.604139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.606782] LustreError: Skipped 1 previous similar message [ 2885.611760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.614158] Lustre: Skipped 1 previous similar message [ 2887.186603] Lustre: server umount lustre-OST0000 complete [ 2900.393815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2900.558556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2900.565413] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2904.641271] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2905.703112] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2905.703426] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2907.674489] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2909.085363] Lustre: Failing over lustre-MDT0000 [ 2909.179899] Lustre: server umount lustre-MDT0000 complete [ 2911.201150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2911.206280] Lustre: Skipped 3 previous similar messages [ 2914.872377] LustreError: 148779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.877792] LustreError: 148779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2916.320951] LustreError: 149574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.328230] LustreError: 149574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2919.992306] LustreError: 151288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.172399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.215513] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.312777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.351780] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2923.751078] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2925.113279] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2927.592252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2927.594255] Lustre: Skipped 1 previous similar message [ 2927.599251] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2927.615395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2931.679944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2931.682784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.688637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2932.704031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2932.706333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2932.712521] Lustre: Skipped 1 previous similar message [ 2934.794346] Lustre: server umount lustre-OST0000 complete [ 2937.824947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.825504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2937.828880] Lustre: Skipped 2 previous similar messages [ 2937.829163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2937.830825] Lustre: Skipped 1 previous similar message [ 2942.300824] Lustre: server umount lustre-MDT0000 complete [ 2942.944307] LustreError: 148784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.953347] LustreError: 148784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2943.721184] LustreError: 148766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784227801 with bad export cookie 7226690098571772270 [ 2943.724097] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.724145] LustreError: 148766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2943.844196] Lustre: server umount lustre-MDT0001 complete [ 2947.945549] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2949.086141] Key type lgssc unregistered [ 2949.223497] LNet: 154389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2949.227617] LNetError: 154389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2949.240264] LNet: Removed LNI 192.168.202.116@tcp [ 2949.589762] Key type .llcrypt unregistered [ 2949.591185] Key type ._llcrypt unregistered [ 2971.324518] Key type ._llcrypt registered [ 2971.326102] Key type .llcrypt registered [ 2971.374314] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2971.839084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2971.853906] alg: No test for adler32 (adler32-zlib) [ 2972.728427] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 2972.836400] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2974.423152] Key type lgssc registered [ 2974.894293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2977.881321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2977.892028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.066557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.674398] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2984.751446] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2986.888025] LustreError: 155934:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2986.893097] LustreError: 155934:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2986.896818] LustreError: 155934:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2987.064449] Lustre: server umount lustre-MDT0000 complete [ 2993.208368] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2997.159822] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3012.945374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.047205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3013.062159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3013.099692] Lustre: lustre-MDT0000: new disk, initializing [ 3013.124411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.132660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3014.718321] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3020.059496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.108950] Lustre: 159232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3020.134766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3020.136569] Lustre: Skipped 1 previous similar message [ 3020.175884] Lustre: lustre-MDT0001: new disk, initializing [ 3020.245563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3020.276877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3020.282293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3022.243320] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3027.413425] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3030.024869] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3032.814068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3032.933257] Lustre: lustre-OST0000: new disk, initializing [ 3032.935785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3032.939827] Lustre: 161200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3032.978360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.867110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3033.870441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3033.911512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3035.319774] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3039.534572] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3042.512766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3043.156388] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:51:41 (1784227901) [ 3219.937445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3219.945639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.960396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3223.523175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3223.523705] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.525804] Lustre: Skipped 1 previous similar message [ 3223.548609] Lustre: Skipped 1 previous similar message [ 3228.643154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3228.646699] Lustre: Skipped 3 previous similar messages [ 3233.763764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3233.769961] Lustre: Skipped 2 previous similar messages [ 3234.272597] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3234.497647] Lustre: server umount lustre-MDT0000 complete [ 3237.255841] LustreError: 159223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784228095 with bad export cookie 5321212823204277802 [ 3237.256685] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3237.264531] LustreError: 159223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3238.880033] LustreError: 159237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.880422] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.885688] LustreError: 159237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3238.911896] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3242.912498] LustreError: 160935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.525284] Lustre: server umount lustre-MDT0001 complete [ 3260.383271] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3260.500543] Lustre: server umount lustre-OST0000 complete [ 3266.359474] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3274.218777] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3299.834021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.994121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3300.015539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3300.083373] Lustre: lustre-MDT0000: new disk, initializing [ 3300.140290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.152540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3304.110995] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3312.458155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.540255] Lustre: 165845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3312.578728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3312.581971] Lustre: Skipped 1 previous similar message [ 3312.642425] Lustre: lustre-MDT0001: new disk, initializing [ 3312.720624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.756701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3312.764991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3315.722574] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3322.594223] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3326.497324] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3330.905504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3331.129575] Lustre: lustre-OST0000: new disk, initializing [ 3331.132686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3331.139099] Lustre: 167813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3331.195745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.694065] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3339.835060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3339.842509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3339.916055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3341.024392] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3344.393307] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3344.584848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3347.518885] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3347.714479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3350.497553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3350.498573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.502874] LustreError: Skipped 1 previous similar message [ 3350.503273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3350.503276] Lustre: Skipped 1 previous similar message [ 3350.537206] Lustre: Skipped 1 previous similar message [ 3363.295449] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3363.440683] Lustre: server umount lustre-OST0000 complete [ 3368.936615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3368.945476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.960292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.968399] Lustre: Skipped 5 previous similar messages [ 3373.299242] Lustre: server umount lustre-MDT0000 complete [ 3375.694640] LustreError: 166650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784228233 with bad export cookie 5321212823204436632 [ 3375.701701] LustreError: 166650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3375.702977] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3375.949454] Lustre: server umount lustre-MDT0001 complete [ 3382.234466] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:57:19 (1784228239) [ 3392.968345] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3400.275157] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3426.077844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.205344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3426.227822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3426.286872] Lustre: lustre-MDT0000: new disk, initializing [ 3426.338187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3426.358431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3429.204336] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3436.992507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.122775] Lustre: 172568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3437.342432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3440.615602] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3447.397924] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3451.667227] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3456.710315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3456.884062] Lustre: 174540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3460.963648] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3463.738883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3463.746547] Lustre: Skipped 1 previous similar message [ 3463.750675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3463.817117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3467.605529] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3474.401090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3474.403191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.405212] LustreError: Skipped 1 previous similar message [ 3474.406206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3474.417125] Lustre: Skipped 2 previous similar messages [ 3474.419748] Lustre: Skipped 2 previous similar messages [ 3478.320454] Lustre: server umount lustre-OST0000 complete [ 3483.617655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3495.391739] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3495.576420] Lustre: server umount lustre-MDT0000 complete [ 3498.305167] LustreError: 172559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784228356 with bad export cookie 5321212823204437514 [ 3498.307123] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3498.312015] LustreError: 172559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3498.635827] Lustre: server umount lustre-MDT0001 complete [ 3504.560896] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:59:22 (1784228362) [ 3515.366590] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3521.945709] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3547.309788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.561161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3547.593372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3547.597234] Lustre: Skipped 3 previous similar messages [ 3547.669594] Lustre: lustre-MDT0000: new disk, initializing [ 3547.674138] Lustre: Skipped 2 previous similar messages [ 3547.774661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.780717] Lustre: Skipped 2 previous similar messages [ 3547.790900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3550.473362] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3557.973558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.045430] Lustre: 178859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3558.197839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3561.303471] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3568.069906] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3571.918377] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3576.826911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3577.065214] Lustre: 180832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3580.990626] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3584.614336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3587.244791] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3594.723062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3594.723801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.728617] LustreError: Skipped 1 previous similar message [ 3594.729995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.739390] Lustre: Skipped 3 previous similar messages [ 3594.751442] Lustre: Skipped 9 previous similar messages [ 3598.206551] Lustre: server umount lustre-OST0000 complete [ 3607.747068] Lustre: server umount lustre-MDT0000 complete [ 3610.082217] LustreError: 179927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.090687] LustreError: 179927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3610.993620] LustreError: 178850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784228468 with bad export cookie 5321212823204438599 [ 3610.994696] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3611.014604] LustreError: 178850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3611.455535] Lustre: server umount lustre-MDT0001 complete [ 3620.787470] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3622.502313] Key type lgssc unregistered [ 3622.674524] LNet: 182700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3622.679336] LNetError: 182700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3622.688971] LNet: Removed LNI 192.168.202.116@tcp [ 3623.255256] Key type .llcrypt unregistered [ 3623.256938] Key type ._llcrypt unregistered [ 3637.092265] Key type ._llcrypt registered [ 3637.093996] Key type .llcrypt registered [ 3637.148259] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3637.559933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3637.625841] alg: No test for adler32 (adler32-zlib) [ 3638.538922] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 3638.678360] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3640.295550] Key type lgssc registered [ 3640.924995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3646.022320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3646.041545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.357197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.721207] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3654.792224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.050183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3657.788917] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3662.563224] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3665.801154] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3669.657772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3669.849382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.623952] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3672.938483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3676.495586] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3694.561573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.564814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3694.565757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3694.575921] Lustre: Skipped 1 previous similar message [ 3699.681670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3699.685855] Lustre: Skipped 2 previous similar messages [ 3700.138677] Lustre: server umount lustre-OST0000 complete [ 3704.805324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.806192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.820865] Lustre: Skipped 1 previous similar message [ 3708.211514] Lustre: server umount lustre-MDT0000 complete [ 3709.921293] LustreError: 184095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.928475] LustreError: 184095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3710.412618] LustreError: 184077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784228568 with bad export cookie 18250211416945952427 [ 3710.413402] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3710.419524] LustreError: 184077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3710.593294] Lustre: server umount lustre-MDT0001 complete [ 3716.593784] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3718.283979] Key type lgssc unregistered [ 3718.454906] LNet: 187651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3718.461823] LNetError: 187651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3718.472580] LNet: Removed LNI 192.168.202.116@tcp [ 3718.948482] Key type .llcrypt unregistered [ 3718.950335] Key type ._llcrypt unregistered [ 3733.469181] Key type ._llcrypt registered [ 3733.470381] Key type .llcrypt registered [ 3733.578430] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3734.303836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3734.319368] alg: No test for adler32 (adler32-zlib) [ 3735.233503] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 3735.418571] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3737.071331] Key type lgssc registered [ 3737.876930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3742.288585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3742.312919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.621151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.025093] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3751.154174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.344122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3753.769121] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3757.750920] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3760.434313] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3764.535486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3764.731432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3767.793404] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3768.809390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3771.205981] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3782.119543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3782.120200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3782.132483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3783.753261] Lustre: lustre-OST0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 3783.759245] Lustre: Skipped 1 previous similar message [ 3785.830894] Lustre: server umount lustre-OST0000 complete [ 3789.943051] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8af2c34a3000.ost_server_uuid 50 [ 3790.900630] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3802.208697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.209194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.219869] Lustre: Skipped 2 previous similar messages [ 3808.376306] Lustre: server umount lustre-MDT0000 complete [ 3808.737426] LustreError: 189044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3808.745232] LustreError: 189044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3810.214433] LustreError: 189026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784228668 with bad export cookie 8158377981523813041 [ 3810.217229] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3810.222960] LustreError: 189026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3810.376321] Lustre: server umount lustre-MDT0001 complete [ 3814.647390] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:04:32 (1784228672) [ 3818.344896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3818.656183] LustreError: 192954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3818.696728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3820.997830] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3824.100863] LustreError: 192955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3826.706045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3826.951882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3829.309612] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3833.189103] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3836.182596] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3839.620857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3839.795361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3842.558780] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3844.901847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3846.085249] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3849.525443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3849.640557] Lustre: lustre-OST0001: new disk, initializing [ 3849.643367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3849.647192] Lustre: 195797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3850.711835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3850.716916] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3850.806582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3852.587398] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3857.593542] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3860.449110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3860.453238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.460229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3860.463644] Lustre: Skipped 3 previous similar messages [ 3861.992742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3865.173337] Lustre: server umount lustre-OST0000 complete [ 3867.116136] LustreError: 194773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3867.134429] LustreError: 194773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3868.654679] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3868.798172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3871.110475] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3871.216472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3872.231566] LustreError: 194773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3872.244953] LustreError: 194773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3882.465969] LustreError: 194773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.469562] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3882.475630] LustreError: 194773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3882.480046] Lustre: Skipped 1 previous similar message [ 3882.485809] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3882.488345] Lustre: Skipped 4 previous similar messages [ 3887.839136] Lustre: server umount lustre-OST0001 complete [ 3892.706883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3892.707734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3892.711294] Lustre: Skipped 2 previous similar messages [ 3892.714379] Lustre: Skipped 2 previous similar messages [ 3895.732334] Lustre: server umount lustre-MDT0000 complete [ 3897.473219] LustreError: 192935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784228755 with bad export cookie 8158377981523814112 [ 3897.474152] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3897.482220] LustreError: 192935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3897.649533] Lustre: server umount lustre-MDT0001 complete [ 3909.114417] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:06:06 (1784228766) [ 3912.341381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.480154] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3912.497883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3912.586722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.591123] Lustre: Skipped 1 previous similar message [ 3914.234816] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3918.536907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.593546] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3918.610382] Lustre: 199209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3920.941348] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3925.074589] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3927.920989] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3931.068239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.160640] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3931.686697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3933.840847] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3937.167905] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3940.012994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3940.061689] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3942.598799] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3945.483303] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3946.983185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3946.984120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.990879] LustreError: Skipped 1 previous similar message [ 3946.992028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3947.009142] Lustre: Skipped 1 previous similar message [ 3947.012965] LustreError: 188356:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 3947.061256] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3952.703653] Lustre: server umount lustre-OST0000 complete [ 3956.283118] LustreError: 201058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3956.290993] LustreError: 201058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3967.462080] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.463696] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3967.472121] LustreError: Skipped 1 previous similar message [ 3969.915464] Lustre: server umount lustre-OST0001 complete [ 3982.818115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3982.821169] Lustre: Skipped 12 previous similar messages [ 3986.399131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3986.477041] Lustre: server umount lustre-MDT0000 complete [ 3988.135679] LustreError: 199199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784228846 with bad export cookie 8158377981523815512 [ 3988.138594] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.143973] LustreError: 199199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3988.298934] Lustre: server umount lustre-MDT0001 complete [ 4000.057706] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:07:37 (1784228857) [ 4007.887167] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4012.744753] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4029.770921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4029.866531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4029.881388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4029.921222] Lustre: lustre-MDT0000: new disk, initializing [ 4029.965085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4029.968452] Lustre: Skipped 3 previous similar messages [ 4029.981468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4031.704388] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4037.290615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.358323] Lustre: 207272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4037.390427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4037.395809] Lustre: Skipped 1 previous similar message [ 4037.456688] Lustre: lustre-MDT0001: new disk, initializing [ 4037.541521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4037.546240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4039.429110] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4044.420619] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4047.240771] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4050.153520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4050.266400] Lustre: lustre-OST0000: new disk, initializing [ 4050.268426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4050.271225] Lustre: 209236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4052.979187] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4056.080117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4056.087659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4056.154275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4057.853642] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4059.890556] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4059.998961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4061.887822] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4061.986162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4066.275964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4066.276920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.280703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4066.285823] Lustre: Skipped 4 previous similar messages [ 4068.961252] Lustre: server umount lustre-OST0000 complete [ 4076.772105] Lustre: server umount lustre-MDT0000 complete [ 4078.342210] LustreError: 207263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784228936 with bad export cookie 8158377981523817262 [ 4078.343950] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4078.346364] LustreError: 207263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4078.480326] Lustre: server umount lustre-MDT0001 complete [ 4081.803728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.984555] LustreError: 211450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4081.993109] LustreError: 211450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4083.635261] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4087.567171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.740341] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4093.201183] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4096.177102] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4099.355170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4099.511890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.514535] Lustre: Skipped 4 previous similar messages [ 4101.897084] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4105.059033] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4107.073225] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4107.199370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4109.357966] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4114.637259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 5 sec [ 4116.960954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4116.967819] LustreError: Skipped 1 previous similar message [ 4121.906212] Lustre: server umount lustre-OST0000 complete [ 4124.775827] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4126.702265] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4151.405180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4154.212933] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4157.921460] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4158.711770] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4159.782400] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4159.788636] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4165.090545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4165.090701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4165.094196] LustreError: Skipped 1 previous similar message [ 4165.098610] Lustre: Skipped 4 previous similar messages [ 4166.109617] Lustre: server umount lustre-OST0000 complete [ 4173.931501] Lustre: server umount lustre-MDT0000 complete [ 4175.328105] LustreError: 212648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.333329] LustreError: 212648:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4175.483909] LustreError: 211431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229033 with bad export cookie 8158377981523818137 [ 4175.486105] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4175.487941] LustreError: 211431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4175.640211] Lustre: server umount lustre-MDT0001 complete [ 4179.295135] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:10:37 (1784229037) [ 4182.089554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.222399] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4188.119731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.025610] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4193.306292] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4195.883539] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4198.622160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4201.270107] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4204.311874] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4206.096509] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4208.241167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4209.926975] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4214.118425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 4216.884932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4216.958419] Lustre: lustre-OST0001: new disk, initializing [ 4216.960694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4216.963854] Lustre: 220851:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4219.290659] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4223.499273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4223.502953] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4223.552368] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4224.094366] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4225.881436] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4225.959849] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4227.566203] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4227.648445] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4228.582236] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4228.585172] Lustre: Skipped 13 previous similar messages [ 4230.464682] Lustre: server umount lustre-OST0001 complete [ 4232.968193] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4233.053531] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4234.849408] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4234.950063] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4259.713183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4259.848708] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4259.853880] Lustre: Skipped 5 previous similar messages [ 4262.522597] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4264.935055] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4264.941377] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4264.945185] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4264.955013] Lustre: Skipped 1 previous similar message [ 4265.776987] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4266.584806] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4270.048464] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4270.053534] LustreError: Skipped 3 previous similar messages [ 4273.484261] Lustre: server umount lustre-OST0001 complete [ 4292.468439] Lustre: server umount lustre-OST0000 complete [ 4295.648327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4295.654478] Lustre: Skipped 9 previous similar messages [ 4300.398196] Lustre: server umount lustre-MDT0000 complete [ 4302.142405] LustreError: 217424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229160 with bad export cookie 8158377981523819082 [ 4302.143233] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4302.151060] LustreError: 217424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4308.358665] Lustre: server umount lustre-MDT0001 complete [ 4318.583535] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:12:56 (1784229176) [ 4321.897098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.036239] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4322.052333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4324.319054] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4328.425413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4328.482652] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4328.509624] Lustre: 226087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4330.488279] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4334.314911] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4337.034103] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4340.302992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4340.418876] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4343.159702] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4346.538218] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4356.495535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4356.528927] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4356.576377] LustreError: 227930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.583965] LustreError: 227930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [ 4358.988714] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4362.409507] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4363.105717] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4365.158703] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4365.264554] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4367.328441] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4367.441874] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4370.097604] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8af2c6035800.ost_server_uuid 50 [ 4370.818734] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8af2c6035800.ost_server_uuid in FULL state after 0 sec [ 4371.327812] Lustre: Permanently deactivating lustre-OST0001 [ 4371.334878] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4372.580135] Lustre: Permanently reactivating lustre-OST0001 [ 4380.042779] Lustre: server umount lustre-OST0001 complete [ 4387.747025] Lustre: server umount lustre-OST0000 complete [ 4395.622662] Lustre: server umount lustre-MDT0000 complete [ 4397.175701] LustreError: 226080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229255 with bad export cookie 8158377981523820356 [ 4397.176334] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4397.181762] LustreError: 226080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4397.330582] Lustre: server umount lustre-MDT0001 complete [ 4407.070423] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:14:25 (1784229265) [ 4409.916364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.033995] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4410.051509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4410.055651] Lustre: Skipped 5 previous similar messages [ 4411.995377] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4415.685818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.758782] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4415.772330] Lustre: 232772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4417.545345] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4420.593644] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4423.037759] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4425.972715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.074319] Lustre: Permanently deactivating lustre-OST0000 [ 4426.080969] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4428.554295] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4431.607740] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4431.876513] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4434.669370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4437.204074] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4438.649636] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4440.328232] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4441.026535] Lustre: Permanently reactivating lustre-OST0000 [ 4445.689743] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4445.693159] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4445.697771] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4445.715306] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4445.722962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4454.371798] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4454.374761] LustreError: Skipped 8 previous similar messages [ 4460.164812] Lustre: server umount lustre-OST0001 complete [ 4467.669257] Lustre: server umount lustre-OST0000 complete [ 4475.433445] Lustre: server umount lustre-MDT0000 complete [ 4477.051152] LustreError: 232765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229335 with bad export cookie 8158377981523822001 [ 4477.054485] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.056106] LustreError: 232765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4477.199492] Lustre: server umount lustre-MDT0001 complete [ 4481.448218] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4482.667639] Key type lgssc unregistered [ 4482.811837] LNet: 237646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4482.814952] LNetError: 237646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4483.877421] LNet: Removed LNI 192.168.202.116@tcp [ 4484.211823] Key type .llcrypt unregistered [ 4484.213271] Key type ._llcrypt unregistered [ 4494.237302] Key type ._llcrypt registered [ 4494.238521] Key type .llcrypt registered [ 4494.287386] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4494.755549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4494.808248] alg: No test for adler32 (adler32-zlib) [ 4495.710094] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 4495.820060] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4497.423244] Key type lgssc registered [ 4497.964408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4502.067465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4502.074873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.262786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4504.907093] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4508.235536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.284881] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4508.297254] Lustre: 239080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4508.302714] Lustre: Permanently deactivating lustre-MDT0001 [ 4508.309778] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4508.451505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4509.106131] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4509.108976] LustreError: 239121:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4509.112991] LustreError: 239121:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4509.116726] LustreError: 239121:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 4509.991130] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4512.865533] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4515.166117] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4517.815297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4517.963966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4519.016626] LustreError: 240931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.023228] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4519.023781] LustreError: 240931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4520.281215] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4523.409947] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4524.514929] LustreError: 240931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4526.110357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4526.213703] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4528.298110] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4528.301077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4528.301149] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4528.647826] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4531.666991] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4533.398785] Lustre: Permanently reactivating lustre-MDT0001 [ 4533.405459] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4533.409078] Lustre: Skipped 2 previous similar messages [ 4537.230463] Lustre: lustre-MDT0001: Client cb78246e-d577-402b-8c3f-836d3a0cb392 (at 192.168.202.16@tcp) reconnecting [ 4539.744942] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4540.321427] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4542.396692] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4542.486436] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4544.054247] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4544.145136] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4545.219804] Lustre: Permanently deactivating lustre-MDT0001 [ 4545.226634] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4545.228394] Lustre: Skipped 2 previous similar messages [ 4548.240957] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4556.258143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4556.260482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4556.263929] Lustre: Skipped 1 previous similar message [ 4561.376716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4561.379291] Lustre: Skipped 6 previous similar messages [ 4562.347227] Lustre: server umount lustre-MDT0000 complete [ 4563.889488] LustreError: 242633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229421 with bad export cookie 16588925505097327615 [ 4563.891947] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4563.894397] LustreError: 242633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4564.017338] Lustre: server umount lustre-MDT0001 complete [ 4571.708841] Lustre: server umount lustre-OST0000 complete [ 4576.008992] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:17:14 (1784229434) [ 4580.063526] Lustre: server umount lustre-OST0001 complete [ 4582.605184] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4586.147417] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4599.656330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.748234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4599.751588] Lustre: Skipped 2 previous similar messages [ 4599.761272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4599.795985] Lustre: lustre-MDT0000: new disk, initializing [ 4599.818625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4599.824913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4601.406696] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4606.036983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.082327] Lustre: 247334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4606.112258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4606.115571] Lustre: Skipped 1 previous similar message [ 4606.158781] Lustre: lustre-MDT0001: new disk, initializing [ 4606.228217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4606.233538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4607.823099] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4611.993165] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4614.336674] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4617.150597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4617.260614] Lustre: lustre-OST0000: new disk, initializing [ 4617.262191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4617.265377] Lustre: 249304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4617.301076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.303682] Lustre: Skipped 1 previous similar message [ 4619.589697] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4620.611961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4620.616508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4620.651497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4623.691367] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4624.386470] LustreError: 247343:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4628.919817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4628.990707] Lustre: lustre-OST0001: new disk, initializing [ 4628.992916] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4628.996493] Lustre: 250417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4631.113354] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4634.479109] LustreError: 247343:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4634.487944] LustreError: 250072:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4635.183945] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4635.770359] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4638.724597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4638.728718] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4638.760776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4644.583110] LustreError: 250072:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4648.516829] Lustre: lustre-OST0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 4648.931936] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4648.933980] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.934885] LustreError: Skipped 1 previous similar message [ 4648.939749] Lustre: Skipped 2 previous similar messages [ 4651.391931] Lustre: server umount lustre-OST0001 complete [ 4654.051352] LustreError: 249287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.057390] LustreError: 249287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4659.173381] LustreError: 249288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.179818] LustreError: 249288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4664.289577] LustreError: 249289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.290393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4664.294265] LustreError: 249289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4664.299228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4664.303260] Lustre: Skipped 3 previous similar messages [ 4669.412443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4669.415104] Lustre: Skipped 1 previous similar message [ 4669.665650] Lustre: server umount lustre-OST0000 complete [ 4672.992541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4672.996130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.001910] Lustre: Skipped 1 previous similar message [ 4677.272497] Lustre: server umount lustre-MDT0000 complete [ 4678.573054] LustreError: 248130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229536 with bad export cookie 16588925505097330422 [ 4678.576232] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4678.576851] LustreError: 248130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4684.729693] Lustre: server umount lustre-MDT0001 complete [ 4688.658887] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4689.732477] Key type lgssc unregistered [ 4689.861369] LNet: 252451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4689.864569] LNetError: 252451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4689.876351] LNet: Removed LNI 192.168.202.116@tcp [ 4690.211001] Key type .llcrypt unregistered [ 4690.212504] Key type ._llcrypt unregistered [ 4704.436226] Key type ._llcrypt registered [ 4704.437494] Key type .llcrypt registered [ 4704.487983] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4705.034895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4705.052808] alg: No test for adler32 (adler32-zlib) [ 4705.912351] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 4706.003261] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4707.583183] Key type lgssc registered [ 4707.976533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4710.618377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4710.624752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.716508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4711.724634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4711.792130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4713.195792] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4716.234881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.272480] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4716.283081] Lustre: 254633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4716.379148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4717.815208] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4720.561334] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4722.457057] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4724.694859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4724.763866] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4724.829095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.832198] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4729.434780] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4732.438387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4778.466422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.467948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4778.471807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4778.474149] LustreError: Skipped 1 previous similar message [ 4778.478217] Lustre: Skipped 1 previous similar message [ 4780.448091] Lustre: server umount lustre-OST0000 complete [ 4782.101271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4787.166115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4787.312693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.480850] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4792.496504] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4798.564845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.573780] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.574594] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4798.582052] Lustre: Skipped 1 previous similar message [ 4829.665481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4829.666539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.670445] LustreError: Skipped 1 previous similar message [ 4829.676870] Lustre: Skipped 1 previous similar message [ 4829.678082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4829.683350] Lustre: Skipped 1 previous similar message [ 4832.677627] Lustre: server umount lustre-OST0000 complete [ 4834.789693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4834.795422] Lustre: Skipped 1 previous similar message [ 4834.798549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4839.392059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4839.395591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.402545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4840.550185] Lustre: server umount lustre-MDT0000 complete [ 4842.153572] LustreError: 255375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229700 with bad export cookie 9371559076160739505 [ 4842.154204] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4842.157911] LustreError: 255375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4842.297690] Lustre: server umount lustre-MDT0001 complete [ 4846.506130] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4847.628796] Key type lgssc unregistered [ 4847.782479] LNet: 260046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4847.785823] LNetError: 260046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4847.795312] LNet: Removed LNI 192.168.202.116@tcp [ 4848.176971] Key type .llcrypt unregistered [ 4848.178512] Key type ._llcrypt unregistered [ 4858.207722] Key type ._llcrypt registered [ 4858.209116] Key type .llcrypt registered [ 4858.250927] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4858.566298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4858.574630] alg: No test for adler32 (adler32-zlib) [ 4859.442540] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 4859.545942] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4861.127165] Key type lgssc registered [ 4861.537186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4864.359058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4864.367532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.520317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4866.936927] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4869.915026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.063724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4871.526941] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4874.219497] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4876.234466] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4878.615978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4878.736909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.736744] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4883.285479] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4890.022205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4905.443206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4905.444060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4905.448971] Lustre: Skipped 1 previous similar message [ 4905.453855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4907.600108] Lustre: server umount lustre-OST0000 complete [ 4910.560704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.561535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4910.564364] Lustre: Skipped 1 previous similar message [ 4910.566466] Lustre: Skipped 1 previous similar message [ 4915.680047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4915.681888] Lustre: Skipped 2 previous similar messages [ 4920.799996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4920.802545] Lustre: Skipped 1 previous similar message [ 4923.361106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4923.435278] Lustre: server umount lustre-MDT0000 complete [ 4924.852308] LustreError: 261418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229782 with bad export cookie 11306639618490375568 [ 4924.855178] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4924.856316] LustreError: 261418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4924.970826] Lustre: server umount lustre-MDT0001 complete [ 4928.873501] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4929.943502] Key type lgssc unregistered [ 4930.080545] LNet: 265762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4930.084303] LNetError: 265762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4930.096297] LNet: Removed LNI 192.168.202.116@tcp [ 4930.411117] Key type .llcrypt unregistered [ 4930.412321] Key type ._llcrypt unregistered [ 4938.122927] Key type ._llcrypt registered [ 4938.124347] Key type .llcrypt registered [ 4938.170153] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4938.570754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4938.623766] alg: No test for adler32 (adler32-zlib) [ 4939.488501] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 4939.579614] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4941.167264] Key type lgssc registered [ 4941.544865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4944.272614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4944.280508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.420967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4946.922920] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4950.067556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.222707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4951.737823] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4954.482936] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4956.670653] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4959.247968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4959.375441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4961.607717] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4964.425882] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4970.662913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4981.219493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4981.221055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4981.225551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4981.227165] LustreError: Skipped 1 previous similar message [ 4981.230987] Lustre: Skipped 1 previous similar message [ 4985.233370] Lustre: server umount lustre-OST0000 complete [ 4991.456722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4991.456775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4991.458156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4991.461627] Lustre: Skipped 2 previous similar messages [ 4992.661886] Lustre: server umount lustre-MDT0000 complete [ 4994.111455] LustreError: 266942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229852 with bad export cookie 5396487110333363313 [ 4994.112850] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4994.116118] LustreError: 266942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4994.236428] Lustre: server umount lustre-MDT0001 complete [ 4998.355532] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4999.485187] Key type lgssc unregistered [ 4999.634607] LNet: 270655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4999.637615] LNetError: 270655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4999.646791] LNet: Removed LNI 192.168.202.116@tcp [ 4999.997828] Key type .llcrypt unregistered [ 4999.999318] Key type ._llcrypt unregistered [ 5010.318679] Key type ._llcrypt registered [ 5010.319977] Key type .llcrypt registered [ 5010.361817] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5010.699248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5010.728072] alg: No test for adler32 (adler32-zlib) [ 5011.587932] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 5011.681118] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5013.263204] Key type lgssc registered [ 5013.665615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5016.610531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5016.619989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.807710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.346669] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5022.481641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5022.609390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5023.977317] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5026.362929] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5028.237756] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5030.608517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5030.724633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5032.786733] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5035.320598] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5042.022040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5053.408433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5053.411589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5053.416897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5057.504784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5057.505119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5057.506892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.511975] Lustre: Skipped 1 previous similar message [ 5059.213925] Lustre: server umount lustre-OST0000 complete [ 5062.624409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.625114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5062.629481] Lustre: Skipped 1 previous similar message [ 5066.515238] Lustre: server umount lustre-MDT0000 complete [ 5067.690234] LustreError: 272074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229925 with bad export cookie 13499656908238913753 [ 5067.692809] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5067.695486] LustreError: 272074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5067.813380] Lustre: server umount lustre-MDT0001 complete [ 5071.518977] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5072.492490] Key type lgssc unregistered [ 5072.621324] LNet: 276372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5072.623445] LNetError: 276372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5072.630282] LNet: Removed LNI 192.168.202.116@tcp [ 5072.922481] Key type .llcrypt unregistered [ 5072.923771] Key type ._llcrypt unregistered [ 5079.702082] Key type ._llcrypt registered [ 5079.703536] Key type .llcrypt registered [ 5079.739961] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5080.094845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5080.128459] alg: No test for adler32 (adler32-zlib) [ 5080.973663] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 5081.051361] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5082.623147] Key type lgssc registered [ 5082.955546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5085.297808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5085.303806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5086.459324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.903190] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5090.938193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.074135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5092.378778] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5094.918897] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5096.677314] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5099.072640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5099.201848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5101.162183] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5103.533790] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5111.526493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5122.018401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5122.019025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.020859] LustreError: Skipped 1 previous similar message [ 5122.024995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5122.027259] Lustre: Skipped 1 previous similar message [ 5123.468987] Lustre: server umount lustre-OST0000 complete [ 5127.137033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.137328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5127.137531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5127.141162] Lustre: Skipped 2 previous similar messages [ 5130.834346] Lustre: server umount lustre-MDT0000 complete [ 5132.047361] LustreError: 277551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784229990 with bad export cookie 9839984770029108748 [ 5132.050422] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5132.050713] LustreError: 277551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5132.166396] Lustre: server umount lustre-MDT0001 complete [ 5135.528066] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5136.518200] Key type lgssc unregistered [ 5136.649338] LNet: 281303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5136.651293] LNetError: 281303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5136.660323] LNet: Removed LNI 192.168.202.116@tcp [ 5136.964802] Key type .llcrypt unregistered [ 5136.965841] Key type ._llcrypt unregistered [ 5140.575314] Key type ._llcrypt registered [ 5140.576202] Key type .llcrypt registered [ 5140.606571] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5145.946269] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5151.983302] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5152.325664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5152.333173] alg: No test for adler32 (adler32-zlib) [ 5153.192803] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 5153.288896] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5154.871161] Key type lgssc registered [ 5155.205235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5165.667066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5165.672745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.758482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5166.768362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5166.804042] Lustre: lustre-MDT0000: new disk, initializing [ 5166.825660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.832753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5168.085406] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5172.157491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5172.200460] Lustre: 284840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5172.222596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5172.224816] Lustre: Skipped 1 previous similar message [ 5172.262527] Lustre: lustre-MDT0001: new disk, initializing [ 5172.295435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5172.307852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5172.311640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5173.743923] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5177.577983] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5179.423386] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5181.691958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5181.787157] Lustre: lustre-OST0000: new disk, initializing [ 5181.789069] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5181.792750] Lustre: 286808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5181.824609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.481525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5183.485653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5183.515388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5183.841440] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5187.511765] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5188.885629] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5188.957330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5190.263842] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5190.336916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5193.697625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5193.698188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.699607] LustreError: Skipped 1 previous similar message [ 5193.704755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5193.707010] Lustre: Skipped 1 previous similar message [ 5197.069555] Lustre: server umount lustre-OST0000 complete [ 5198.816311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.816807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5198.820240] Lustre: Skipped 2 previous similar messages [ 5198.825105] Lustre: Skipped 1 previous similar message [ 5203.936058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5203.938106] Lustre: Skipped 1 previous similar message [ 5204.370656] Lustre: server umount lustre-MDT0000 complete [ 5205.571084] LustreError: 284831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230063 with bad export cookie 14500215191134085399 [ 5205.574066] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5205.574918] LustreError: 284831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5205.685875] Lustre: server umount lustre-MDT0001 complete [ 5208.218970] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:27:46 (1784230066) [ 5210.260811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.416808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.638348] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5214.398478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.514573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5215.826724] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5218.187271] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5219.928407] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5221.994813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5223.905413] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5226.151681] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5239.775819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5239.778433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.782540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5240.802232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5245.713386] Lustre: server umount lustre-OST0000 complete [ 5250.016880] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5250.019168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.022139] Lustre: Skipped 1 previous similar message [ 5250.024122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5250.026723] Lustre: Skipped 2 previous similar messages [ 5253.208088] Lustre: server umount lustre-MDT0000 complete [ 5254.449975] LustreError: 289196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230112 with bad export cookie 14500215191134086267 [ 5254.451457] LustreError: 289953:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5254.453374] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5254.454035] LustreError: 289196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5254.564525] Lustre: server umount lustre-MDT0001 complete [ 5258.108671] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5259.114433] Key type lgssc unregistered [ 5259.251386] LNet: 292769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5259.254796] LNetError: 292769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5259.264365] LNet: Removed LNI 192.168.202.116@tcp [ 5259.597681] Key type .llcrypt unregistered [ 5259.599380] Key type ._llcrypt unregistered [ 5267.961449] Key type ._llcrypt registered [ 5267.962342] Key type .llcrypt registered [ 5268.002380] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5273.601384] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5273.973980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5274.011089] alg: No test for adler32 (adler32-zlib) [ 5274.855061] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 5274.933135] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5276.511137] Key type lgssc registered [ 5276.846493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5290.744068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5290.749799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5291.831148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5291.841655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5291.875842] Lustre: lustre-MDT0000: new disk, initializing [ 5291.896305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5291.902531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5293.227222] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5297.291946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.334266] Lustre: 296565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5297.362198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5297.364768] Lustre: Skipped 1 previous similar message [ 5297.399801] Lustre: lustre-MDT0001: new disk, initializing [ 5297.430924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5297.441941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5297.444936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5298.763898] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5302.415633] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5304.084860] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5306.275240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5306.376163] Lustre: lustre-OST2710: new disk, initializing [ 5306.378450] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5306.381827] Lustre: 298531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5306.411029] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5308.199861] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5311.785856] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5313.914204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5313.968287] Lustre: lustre-OST03e8: new disk, initializing [ 5313.970123] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5313.972799] Lustre: 299581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5313.999310] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5315.819207] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5316.164154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5316.168734] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5316.230845] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5316.238559] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5318.223056] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5320.794440] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5320.862386] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5322.086082] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5322.152221] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5323.393679] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5323.458479] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5324.674499] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5324.740751] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5326.817532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.818801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5326.822069] Lustre: Skipped 1 previous similar message [ 5326.824708] Lustre: Skipped 3 previous similar messages [ 5331.875493] Lustre: server umount lustre-MDT0000 complete [ 5333.089962] LustreError: 296558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230191 with bad export cookie 4944821430429394626 [ 5333.093268] LustreError: 296558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5333.093549] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.207764] Lustre: server umount lustre-MDT0001 complete [ 5344.259073] Lustre: server umount lustre-OST2710 complete [ 5355.682553] Lustre: server umount lustre-OST03e8 complete [ 5361.192502] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5364.394449] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5375.609667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.681230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5375.691975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5375.728119] Lustre: lustre-MDT0000: new disk, initializing [ 5375.749336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.755858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5375.758087] Lustre: Skipped 1 previous similar message [ 5376.962781] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5380.835913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.872524] Lustre: 304829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5380.960273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5380.962811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5380.966672] Lustre: Skipped 1 previous similar message [ 5382.206926] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5385.836561] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5387.603027] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5389.784290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5389.874903] Lustre: lustre-OST0000: new disk, initializing [ 5389.876087] Lustre: Skipped 1 previous similar message [ 5389.877432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5389.878837] Lustre: Skipped 2 previous similar messages [ 5389.880608] Lustre: 306793:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5389.904422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5389.907296] Lustre: Skipped 1 previous similar message [ 5390.206298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5390.210312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5390.248166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5391.917039] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5395.518682] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5396.782416] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5396.854752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5398.295894] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5398.372749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5400.547160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5400.547167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5400.547780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5400.551718] Lustre: Skipped 3 previous similar messages [ 5400.553994] LustreError: Skipped 1 previous similar message [ 5405.666339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5405.668534] Lustre: Skipped 2 previous similar messages [ 5410.785495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5410.787658] Lustre: Skipped 1 previous similar message [ 5413.343141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5413.400039] Lustre: server umount lustre-OST0000 complete [ 5415.904619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.905585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5415.909027] Lustre: Skipped 1 previous similar message [ 5426.144126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5426.146523] Lustre: Skipped 4 previous similar messages [ 5429.215159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5429.274221] Lustre: server umount lustre-MDT0000 complete [ 5430.479180] LustreError: 304819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230288 with bad export cookie 4944821430429396194 [ 5430.482127] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5430.483104] LustreError: 304819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5430.607057] Lustre: server umount lustre-MDT0001 complete [ 5433.271634] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:31:31 (1784230291) [ 5433.726066] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5434.261020] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:31:32 (1784230292) [ 5435.221644] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5443.603753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.675026] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5443.680645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5443.730136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.964447] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5447.617239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5447.646481] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5447.655567] Lustre: 310647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5448.982662] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5464.546017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5464.614755] Lustre: Denying initial registration attempt from nid 192.168.202.116@tcp, specified as failover [ 5464.617966] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5464.622278] LustreError: 312101:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5464.624875] LustreError: 312101:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5464.627418] LustreError: 312101:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5464.647628] Lustre: server umount lustre-OST0000 complete [ 5464.648974] LustreError: 312101:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5468.127850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5468.130062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.133725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5472.914272] Lustre: server umount lustre-MDT0000 complete [ 5474.159330] LustreError: 310639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230332 with bad export cookie 4944821430429397062 [ 5474.162281] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5474.165474] LustreError: 310639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5474.273993] Lustre: server umount lustre-MDT0001 complete [ 5477.986355] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5479.044759] Key type lgssc unregistered [ 5479.168415] LNet: 313220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5479.170968] LNetError: 313220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5479.185330] LNet: Removed LNI 192.168.202.116@tcp [ 5479.518062] Key type .llcrypt unregistered [ 5479.519134] Key type ._llcrypt unregistered [ 5483.330662] Key type ._llcrypt registered [ 5483.332220] Key type .llcrypt registered [ 5483.377142] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5483.801719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5483.819339] alg: No test for adler32 (adler32-zlib) [ 5484.676941] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 5484.779229] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5486.367181] Key type lgssc registered [ 5486.780831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5510.199821] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5513.014147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5513.019332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5514.093234] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5514.098719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5514.146627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.404458] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5518.058171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.088616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5518.098431] Lustre: 315959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5518.183175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5519.441103] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5521.680852] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5523.323122] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5538.529778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5538.592711] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5538.657936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.497016] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5542.790278] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5543.513927] Lustre: server umount lustre-OST0000 complete [ 5547.503290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.504346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5547.507741] Lustre: Skipped 1 previous similar message [ 5552.608177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5552.610302] Lustre: Skipped 2 previous similar messages [ 5557.728072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5557.730456] Lustre: Skipped 1 previous similar message [ 5559.263143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5559.320395] Lustre: server umount lustre-MDT0000 complete [ 5560.621478] LustreError: 315949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230418 with bad export cookie 15971956504704442201 [ 5560.625409] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5560.626085] LustreError: 315949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5560.753273] Lustre: server umount lustre-MDT0001 complete [ 5564.513856] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5565.524661] Key type lgssc unregistered [ 5565.650464] LNet: 319520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5565.652913] LNetError: 319520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5565.660659] LNet: Removed LNI 192.168.202.116@tcp [ 5565.964650] Key type .llcrypt unregistered [ 5565.966025] Key type ._llcrypt unregistered [ 5574.416778] Key type ._llcrypt registered [ 5574.417634] Key type .llcrypt registered [ 5574.448702] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5574.694826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5574.712318] alg: No test for adler32 (adler32-zlib) [ 5575.562751] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 5575.641870] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5577.223150] Key type lgssc registered [ 5577.562945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5580.009291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5580.016253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.146512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5582.401486] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5585.169626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5585.301294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5586.603612] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5588.962690] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5590.724244] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5594.913723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5595.017567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5596.892911] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5599.128845] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5614.655262] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 5616.095860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5616.098015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.101673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5616.609555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.614254] Lustre: Skipped 1 previous similar message [ 5617.752069] Lustre: server umount lustre-MDT0000 complete [ 5619.051382] LustreError: 320893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230477 with bad export cookie 8902748991075675425 [ 5619.054847] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5619.056685] LustreError: 320893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5619.769256] LustreError: 322352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.775587] LustreError: 322352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5619.779395] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 5619.781684] Lustre: Skipped 2 previous similar messages [ 5621.728142] LustreError: 320906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.728470] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.735781] LustreError: 320907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5624.736260] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5624.744468] Lustre: Skipped 1 previous similar message [ 5625.208856] Lustre: server umount lustre-MDT0001 complete [ 5627.403531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5630.128375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.255646] LustreError: 320224:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007d0ba475 x1870901210538752/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5644.357607] LustreError: 322743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5644.365260] LustreError: 322743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5644.378320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5644.394589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5645.639105] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5648.432708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.578395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5649.636745] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5649.930333] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5652.228762] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5654.758898] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5655.793772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5657.687178] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5658.244346] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5659.616087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5659.618530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.623537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5659.625874] Lustre: Skipped 1 previous similar message [ 5665.624745] Lustre: server umount lustre-OST0000 complete [ 5669.343765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5669.346291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.350508] Lustre: Skipped 1 previous similar message [ 5669.352681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5669.354789] Lustre: Skipped 4 previous similar messages [ 5673.051257] Lustre: server umount lustre-MDT0000 complete [ 5674.309271] LustreError: 326162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230532 with bad export cookie 8902748991075682579 [ 5674.312270] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5674.313260] LustreError: 326162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5674.428832] Lustre: server umount lustre-MDT0001 complete [ 5678.055719] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5679.032462] Key type lgssc unregistered [ 5679.161363] LNet: 327201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5679.164205] LNetError: 327201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5679.177287] LNet: Removed LNI 192.168.202.116@tcp [ 5679.455363] Key type .llcrypt unregistered [ 5679.456885] Key type ._llcrypt unregistered [ 5687.956113] Key type ._llcrypt registered [ 5687.956898] Key type .llcrypt registered [ 5687.992100] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5688.391020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5688.403789] alg: No test for adler32 (adler32-zlib) [ 5689.249138] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 5689.335237] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5690.911136] Key type lgssc registered [ 5691.248306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5693.740690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5693.746851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.882252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.236903] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5699.087980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.222901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5700.539951] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5702.927225] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5704.770551] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5709.791990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5709.793066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.795232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5709.799115] Lustre: Skipped 1 previous similar message [ 5709.802474] Lustre: Skipped 1 previous similar message [ 5710.816307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5710.818983] Lustre: Skipped 1 previous similar message [ 5715.938024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5715.940157] Lustre: Skipped 1 previous similar message [ 5720.031120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5720.090572] Lustre: server umount lustre-MDT0000 complete [ 5721.056174] LustreError: 329323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.062564] LustreError: 329323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5721.299339] LustreError: 328575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230579 with bad export cookie 5998029550997808143 [ 5721.302262] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5721.303815] LustreError: 328575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5721.414865] Lustre: server umount lustre-MDT0001 complete [ 5725.975412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.087938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5726.098934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5726.182202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.549406] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5730.321044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.349422] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5730.359725] Lustre: 331181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5730.445145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5731.717729] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5733.985873] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5735.724723] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5740.193920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5740.267261] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5740.272350] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5740.274772] LustreError: 333062:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5740.276795] LustreError: 333062:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5740.278397] LustreError: 333062:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5740.301846] Lustre: server umount lustre-OST0000 complete [ 5740.303059] LustreError: 333062:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5744.994229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.060967] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5745.119595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5746.959400] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5749.199077] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5751.568720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5751.596682] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5751.622401] Lustre: lustre-OST0001: new disk, initializing [ 5751.624344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5751.627094] Lustre: 334592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5753.509171] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5754.892878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5754.896809] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5754.923085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5754.945664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5755.811587] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5759.970844] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5759.973135] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.973272] LustreError: Skipped 1 previous similar message [ 5759.977683] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5759.980127] Lustre: Skipped 1 previous similar message [ 5762.817573] Lustre: server umount lustre-OST0001 complete [ 5765.090635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5765.091143] LustreError: 333529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.091385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.091394] Lustre: Skipped 1 previous similar message [ 5765.092241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5765.105097] LustreError: 333529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5770.211872] LustreError: 335168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.217348] LustreError: 335168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5775.332144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.333334] LustreError: 333530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.333676] Lustre: Skipped 3 previous similar messages [ 5775.337725] LustreError: 333530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5778.399167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5778.450838] Lustre: server umount lustre-OST0000 complete [ 5780.448711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.453028] Lustre: Skipped 2 previous similar messages [ 5785.810545] Lustre: server umount lustre-MDT0000 complete [ 5786.922298] LustreError: 331172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230644 with bad export cookie 5998029550997808682 [ 5786.925498] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5786.926028] LustreError: 331172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5787.033819] Lustre: server umount lustre-MDT0001 complete [ 5790.375527] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5791.356620] Key type lgssc unregistered [ 5791.486366] LNet: 336553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5791.488658] LNetError: 336553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5791.497279] LNet: Removed LNI 192.168.202.116@tcp [ 5791.822903] Key type .llcrypt unregistered [ 5791.824347] Key type ._llcrypt unregistered [ 5822.351209] Key type ._llcrypt registered [ 5822.352153] Key type .llcrypt registered [ 5822.385816] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5828.256855] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5828.590199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5828.638096] alg: No test for adler32 (adler32-zlib) [ 5829.486479] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 5829.575881] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5831.151210] Key type lgssc registered [ 5831.533573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5842.060489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5842.065027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.144816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5843.154522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5843.191178] Lustre: lustre-MDT0000: new disk, initializing [ 5843.213596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.220647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5844.484857] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5848.300601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5848.338747] Lustre: 341456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5848.359531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5848.361400] Lustre: Skipped 1 previous similar message [ 5848.397364] Lustre: lustre-MDT0001: new disk, initializing [ 5848.425755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5848.436664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5848.440195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5849.607976] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5853.258488] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5855.097553] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5857.388206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.497634] Lustre: lustre-OST0000: new disk, initializing [ 5857.499264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5857.501871] Lustre: 343421:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5857.534336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5858.105139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5858.107985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5858.150455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5859.491188] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5863.216525] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5864.740712] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5864.827939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5866.167857] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5866.234551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5868.514236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.514809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.518442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.521758] Lustre: Skipped 1 previous similar message [ 5873.634043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5873.636094] Lustre: Skipped 2 previous similar messages [ 5878.753855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5878.755775] Lustre: Skipped 1 previous similar message [ 5881.311113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5881.365278] Lustre: server umount lustre-OST0000 complete [ 5883.872559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.873473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5883.876742] Lustre: Skipped 1 previous similar message [ 5888.793221] Lustre: server umount lustre-MDT0000 complete [ 5890.023184] LustreError: 342248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230748 with bad export cookie 10351208007159485102 [ 5890.026676] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.027549] LustreError: 342248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5890.145957] Lustre: server umount lustre-MDT0001 complete [ 5892.923090] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:39:11 (1784230751) [ 5895.333314] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:39:13 (1784230753) [ 5898.236477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.390627] LustreError: 346175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5898.398429] LustreError: 346175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5898.417350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5899.806223] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5902.589728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5902.705798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5904.169422] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5906.642553] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5908.535090] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5910.789845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5910.927638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5912.906334] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5915.463286] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5929.342244] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5929.925753] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5930.499884] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5931.129676] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5931.823436] Lustre: Failing over lustre-MDT0000 [ 5931.971405] Lustre: server umount lustre-MDT0000 complete [ 5933.537583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5933.538488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.540489] LustreError: Skipped 1 previous similar message [ 5933.545742] LustreError: 346175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.550995] LustreError: 346175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5936.190332] LustreError: 346171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.656795] LustreError: 346176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.661507] LustreError: 346176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5943.777379] LustreError: 346903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.785687] LustreError: 346903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5944.863878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.911127] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5945.020051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5945.052867] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5946.425875] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5946.450681] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5950.440375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5950.452256] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5950.469280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5952.380276] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5952.875626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5954.724607] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5955.425836] Lustre: Failing over lustre-MDT0000 [ 5955.554086] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.555850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5955.560627] Lustre: Skipped 4 previous similar messages [ 5955.560991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5955.564181] Lustre: Skipped 3 previous similar messages [ 5955.644637] Lustre: server umount lustre-MDT0000 complete [ 5956.672794] LustreError: 348665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.700095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.744080] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5968.885236] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5970.378282] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5972.024363] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5973.991902] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5973.994258] Lustre: Skipped 2 previous similar messages [ 5974.002819] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5974.020216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5975.677909] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5976.135506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5979.105307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5979.105616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5979.106767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5979.113756] Lustre: Skipped 1 previous similar message [ 5984.473486] Lustre: server umount lustre-MDT0000 complete [ 5985.686894] LustreError: 346157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230843 with bad export cookie 10351208007159487783 [ 5985.689594] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5985.690459] LustreError: 346157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5985.782765] Lustre: server umount lustre-MDT0001 complete [ 5996.186603] Lustre: server umount lustre-OST0000 complete [ 6001.339903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.499280] LustreError: 352527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.506191] LustreError: 352527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6001.523726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.526270] Lustre: Skipped 1 previous similar message [ 6002.916865] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6005.809919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.376583] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6009.838446] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6011.730447] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6014.041829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6016.139740] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6018.640244] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6026.470473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6030.816067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6030.818500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.823542] Lustre: Skipped 1 previous similar message [ 6030.825314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6030.827776] Lustre: Skipped 5 previous similar messages [ 6032.785916] Lustre: server umount lustre-OST0000 complete [ 6036.188330] Lustre: server umount lustre-MDT0000 complete [ 6036.961213] LustreError: 352528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.966020] LustreError: 352528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6037.463328] LustreError: 352509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230895 with bad export cookie 10351208007159488504 [ 6037.466543] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6037.468383] LustreError: 352509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6037.585419] Lustre: server umount lustre-MDT0001 complete [ 6041.194784] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6042.231599] Key type lgssc unregistered [ 6042.354345] LNet: 356077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6042.357100] LNetError: 356077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6042.365308] LNet: Removed LNI 192.168.202.116@tcp [ 6042.676512] Key type .llcrypt unregistered [ 6042.677765] Key type ._llcrypt unregistered [ 6052.337336] Key type ._llcrypt registered [ 6052.338191] Key type .llcrypt registered [ 6052.379344] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6052.866524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6052.893575] alg: No test for adler32 (adler32-zlib) [ 6053.741366] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 6053.821647] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6055.399096] Key type lgssc registered [ 6055.701723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6057.991526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6057.996342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6059.135295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.323469] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6063.080972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.206221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6064.633893] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6067.207958] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6069.118326] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6071.431766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6071.574858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.537711] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6075.933090] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6084.899463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6090.209417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.210497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6090.213389] Lustre: Skipped 1 previous similar message [ 6095.329455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6095.331714] Lustre: Skipped 4 previous similar messages [ 6096.219312] Lustre: server umount lustre-MDT0000 complete [ 6097.505117] LustreError: 357550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784230955 with bad export cookie 4100047392206721178 [ 6097.508153] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6097.509146] LustreError: 357550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6097.635825] Lustre: server umount lustre-MDT0001 complete [ 6108.122149] Lustre: server umount lustre-OST0000 complete [ 6112.751605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.917182] LustreError: 361295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.923462] LustreError: 361295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6112.941457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.355074] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6117.411640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.537866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6118.955832] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6121.464663] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6123.434428] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6125.772052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6125.904992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6127.936819] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6130.362854] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6137.191292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6148.576664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6148.580046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.583077] Lustre: Skipped 1 previous similar message [ 6148.584656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6152.672858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6152.675961] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.680257] Lustre: Skipped 1 previous similar message [ 6153.945811] Lustre: server umount lustre-MDT0000 complete [ 6155.253925] LustreError: 361277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231013 with bad export cookie 4100047392206722277 [ 6155.257870] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6155.261418] LustreError: 361277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6155.388150] Lustre: server umount lustre-MDT0001 complete [ 6167.002316] Lustre: server umount lustre-OST0000 complete [ 6172.129320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.301436] LustreError: 365222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.325626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6173.664711] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6176.557353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.101503] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6180.592522] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6182.460639] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6184.839310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6186.936220] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6189.413308] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6197.285583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6201.311765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6201.313761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.319101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6201.320640] Lustre: Skipped 2 previous similar messages [ 6202.336690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6203.152952] Lustre: server umount lustre-OST0000 complete [ 6206.488109] Lustre: server umount lustre-MDT0000 complete [ 6207.457203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6207.457428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6207.458160] LustreError: 365223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6207.461127] Lustre: Skipped 2 previous similar messages [ 6207.758659] LustreError: 365202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231065 with bad export cookie 4100047392206723285 [ 6207.761560] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6207.763034] LustreError: 365202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6207.898100] Lustre: server umount lustre-MDT0001 complete [ 6211.433368] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6212.534264] Key type lgssc unregistered [ 6212.669404] LNet: 368770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6212.671674] LNetError: 368770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6212.680377] LNet: Removed LNI 192.168.202.116@tcp [ 6213.026520] Key type .llcrypt unregistered [ 6213.028408] Key type ._llcrypt unregistered [ 6222.543908] Key type ._llcrypt registered [ 6222.544858] Key type .llcrypt registered [ 6222.584598] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6223.111959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6223.127102] alg: No test for adler32 (adler32-zlib) [ 6223.990654] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 6224.090275] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6225.671128] Key type lgssc registered [ 6226.023144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6228.647778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6228.653282] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.655217] LustreError: 370186:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6228.662918] LustreError: 370186:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6228.665127] LustreError: 370186:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6228.667376] LustreError: 370186:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6228.669759] LustreError: 370186:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6231.138660] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6231.141084] LustreError: 370583:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6231.150300] LustreError: 370583:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6231.152687] LustreError: 370583:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6231.154854] LustreError: 370583:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6231.157293] LustreError: 370583:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6237.850801] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6238.880414] Key type lgssc unregistered [ 6239.010285] LNet: 371585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6239.012621] LNetError: 371585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6239.020328] LNet: Removed LNI 192.168.202.116@tcp [ 6239.321643] Key type .llcrypt unregistered [ 6239.322736] Key type ._llcrypt unregistered [ 6245.307499] Key type ._llcrypt registered [ 6245.308496] Key type .llcrypt registered [ 6245.344527] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6251.434799] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6251.890933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6251.908841] alg: No test for adler32 (adler32-zlib) [ 6252.757427] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 6252.852665] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6254.431099] Key type lgssc registered [ 6254.803283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6266.144315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6266.150440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6267.247658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6267.257541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6267.300862] Lustre: lustre-MDT0000: new disk, initializing [ 6267.321343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6267.326789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6268.651688] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6272.728291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.773360] Lustre: 374713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6272.797363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6272.799498] Lustre: Skipped 1 previous similar message [ 6272.842825] Lustre: lustre-MDT0001: new disk, initializing [ 6272.875315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6272.887277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6272.891213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6274.213504] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6277.982902] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6279.832663] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6282.077073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.196444] Lustre: lustre-OST0000: new disk, initializing [ 6282.198841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6282.201538] Lustre: 376681:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6282.230400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6284.107871] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6287.769218] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6288.888883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6288.891841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6288.923304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6289.137881] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6289.205703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6290.592364] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6290.676845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6293.986053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6293.986464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6293.988661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6293.991015] Lustre: Skipped 1 previous similar message [ 6297.433522] Lustre: server umount lustre-OST0000 complete [ 6299.104704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.105655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.107849] Lustre: Skipped 1 previous similar message [ 6299.109263] Lustre: Skipped 1 previous similar message [ 6304.224162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6304.226882] Lustre: Skipped 2 previous similar messages [ 6304.794333] Lustre: server umount lustre-MDT0000 complete [ 6306.059335] LustreError: 374707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231164 with bad export cookie 4327232440507295050 [ 6306.063346] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6306.064264] LustreError: 374707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6306.200448] Lustre: server umount lustre-MDT0001 complete [ 6308.898927] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6309.433598] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:46:07 (1784231167) [ 6311.629932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.799787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6313.197086] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6316.042388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.159163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6317.540383] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6319.950889] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6321.819476] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6324.027019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6326.112851] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6328.571679] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6330.826412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6330.901831] Lustre: lustre-OST0001: new disk, initializing [ 6330.903685] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6330.906320] Lustre: 381966:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6330.940047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6330.943411] Lustre: Skipped 1 previous similar message [ 6332.967878] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6335.508524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6335.512657] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6335.559553] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6336.755065] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6340.576204] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6340.576438] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.580126] LustreError: Skipped 2 previous similar messages [ 6340.586231] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6340.589590] Lustre: Skipped 1 previous similar message [ 6344.700436] Lustre: server umount lustre-OST0001 complete [ 6346.724829] LustreError: 382542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6346.731121] LustreError: 382542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6351.845352] LustreError: 380956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.850608] LustreError: 380956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6356.962849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.962952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6356.964097] LustreError: 380954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.965581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6356.965587] Lustre: Skipped 3 previous similar messages [ 6356.966374] Lustre: Skipped 2 previous similar messages [ 6356.983317] LustreError: Skipped 1 previous similar message [ 6362.082885] LustreError: 380955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6362.089262] LustreError: 380955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6362.826823] Lustre: server umount lustre-OST0000 complete [ 6367.201208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.202250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6367.206516] Lustre: Skipped 1 previous similar message [ 6367.210877] Lustre: Skipped 4 previous similar messages [ 6370.268071] Lustre: server umount lustre-MDT0000 complete [ 6371.506356] LustreError: 379116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231229 with bad export cookie 4327232440507295918 [ 6371.510097] LustreError: 379116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6371.510685] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6377.724673] Lustre: server umount lustre-MDT0001 complete [ 6381.484850] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6382.612752] Key type lgssc unregistered [ 6382.746473] LNet: 383948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6382.749058] LNetError: 383948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6382.761337] LNet: Removed LNI 192.168.202.116@tcp [ 6383.084047] Key type .llcrypt unregistered [ 6383.086082] Key type ._llcrypt unregistered [ 6400.950209] Key type ._llcrypt registered [ 6400.951191] Key type .llcrypt registered [ 6400.985188] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6401.415189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6401.428071] alg: No test for adler32 (adler32-zlib) [ 6402.286199] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 6402.384364] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6403.967193] Key type lgssc registered [ 6404.432564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6407.087675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6407.099595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6408.208522] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6408.214858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6408.274119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6409.594546] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6412.434159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.470134] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6412.482433] Lustre: 386721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6412.581849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6413.987400] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6416.406638] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6418.260317] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6419.936550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.937379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6419.939654] Lustre: Skipped 1 previous similar message [ 6425.060108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6425.061936] Lustre: Skipped 2 previous similar messages [ 6430.176171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6430.177957] Lustre: Skipped 1 previous similar message [ 6433.247109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6433.310203] Lustre: server umount lustre-MDT0000 complete [ 6434.528253] LustreError: 386713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231292 with bad export cookie 13841782269063205193 [ 6434.531471] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6434.532508] LustreError: 386713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6434.638025] Lustre: server umount lustre-MDT0001 complete [ 6437.559230] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:48:15 (1784231295) [ 6440.020406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.199155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.456048] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6444.577252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.711427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6446.054955] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6448.516787] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6450.351419] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6452.616142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6452.687767] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6452.746548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.550095] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6456.896286] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6466.070654] Lustre: Permanently deactivating lustre-OST0000 [ 6466.075683] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6475.247802] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6475.917936] LustreError: 392363:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6475.921377] LustreError: 392363:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 6475.923405] LustreError: 392363:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6476.549420] LustreError: 392411:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 6476.554592] LustreError: 392411:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6483.278587] Lustre: server umount lustre-OST0000 complete [ 6484.218542] LustreError: 392660:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 6484.221246] LustreError: 392660:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6485.984051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6485.986398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6485.990827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6488.545333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.665112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6493.669660] Lustre: Skipped 3 previous similar messages [ 6499.295091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6499.353669] Lustre: server umount lustre-MDT0000 complete [ 6500.571134] LustreError: 389214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231358 with bad export cookie 13841782269063205760 [ 6500.574166] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6500.575425] LustreError: 389214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6510.106316] Lustre: server umount lustre-MDT0001 complete [ 6512.996813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.394572] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6515.331699] LustreError: 393818:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6517.466263] Lustre: 394156:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.116@tcp [ 6518.149192] Lustre: server umount MGS complete [ 6521.894155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.057332] LustreError: 394879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6522.061835] LustreError: 394879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6522.067043] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6522.071364] Lustre: Skipped 1 previous similar message [ 6522.096396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.420293] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6526.203257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6526.304393] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6527.642904] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6530.073658] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6531.859765] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6532.532466] Lustre: Permanently reactivating lustre-OST0000 [ 6532.537992] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6532.539876] Lustre: Skipped 2 previous similar messages [ 6561.759123] LustreError: 396804:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6564.738029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6564.852830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6564.856136] Lustre: Skipped 1 previous similar message [ 6566.828546] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6569.297414] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6581.217464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6581.217976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.219495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6581.219499] Lustre: Skipped 2 previous similar messages [ 6581.220119] LustreError: Skipped 1 previous similar message [ 6581.227901] Lustre: Skipped 1 previous similar message [ 6583.183926] Lustre: server umount lustre-OST0000 complete [ 6586.336435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6586.341416] Lustre: Skipped 1 previous similar message [ 6586.519597] Lustre: server umount lustre-MDT0000 complete [ 6587.749313] LustreError: 394860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231445 with bad export cookie 13841782269063206796 [ 6587.752858] LustreError: 394860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6587.752984] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6587.864199] Lustre: server umount lustre-MDT0001 complete [ 6591.593036] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6592.638975] Key type lgssc unregistered [ 6592.767387] LNet: 399695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6592.769976] LNetError: 399695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6592.777333] LNet: Removed LNI 192.168.202.116@tcp [ 6593.093119] Key type .llcrypt unregistered [ 6593.094522] Key type ._llcrypt unregistered [ 6598.962154] Key type ._llcrypt registered [ 6598.963092] Key type .llcrypt registered [ 6599.000121] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6605.448273] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6605.993496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6606.005164] alg: No test for adler32 (adler32-zlib) [ 6606.869745] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 6606.968438] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6608.551225] Key type lgssc registered [ 6608.959201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6620.616352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6620.621884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.714186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6621.725516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6621.765297] Lustre: lustre-MDT0000: new disk, initializing [ 6621.788474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.795400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6623.222226] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6627.357825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.403493] Lustre: 402825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6627.431264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6627.433287] Lustre: Skipped 1 previous similar message [ 6627.476554] Lustre: lustre-MDT0001: new disk, initializing [ 6627.520512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6627.533539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6627.537303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6628.957418] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6632.683409] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6634.561520] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6636.836483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.964248] Lustre: lustre-OST0000: new disk, initializing [ 6636.965794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6636.967957] Lustre: 404792:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6637.001172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6638.935452] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6639.161506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6639.165191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6639.198276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6642.652434] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6644.101445] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6644.181929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6645.592134] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6645.668204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6649.315487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6649.315621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6649.320048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6649.323009] Lustre: Skipped 1 previous similar message [ 6652.431406] Lustre: server umount lustre-OST0000 complete [ 6654.432782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6654.433565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6654.437124] Lustre: Skipped 2 previous similar messages [ 6654.439905] Lustre: Skipped 1 previous similar message [ 6659.552099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6659.553750] Lustre: Skipped 1 previous similar message [ 6659.806287] Lustre: server umount lustre-MDT0000 complete [ 6661.034782] LustreError: 402816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231519 with bad export cookie 12793460451290049820 [ 6661.037204] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6661.040174] LustreError: 402816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6661.164557] Lustre: server umount lustre-MDT0001 complete [ 6663.969359] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6664.543814] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:52:02 (1784231522) [ 6666.853897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6667.036393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6668.347196] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6671.154621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6671.277852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6672.693343] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6675.130488] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6676.961535] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6679.164399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6679.299786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.248839] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6683.723121] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6684.355595] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6695.052718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6696.929588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6696.931728] LustreError: Skipped 1 previous similar message [ 6696.932901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.936721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6697.953299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6702.614836] Lustre: server umount lustre-OST0000 complete [ 6707.167761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6707.169770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.173091] Lustre: Skipped 1 previous similar message [ 6707.175162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6707.176688] Lustre: Skipped 2 previous similar messages [ 6709.972959] Lustre: server umount lustre-MDT0000 complete [ 6711.220400] LustreError: 407230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231569 with bad export cookie 12793460451290050688 [ 6711.224534] LustreError: 407230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6711.225531] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6711.347802] Lustre: server umount lustre-MDT0001 complete [ 6715.091860] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6716.152798] Key type lgssc unregistered [ 6716.290403] LNet: 410990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6716.292653] LNetError: 410990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6716.300314] LNet: Removed LNI 192.168.202.116@tcp [ 6716.608480] Key type .llcrypt unregistered [ 6716.609599] Key type ._llcrypt unregistered [ 6724.795098] Key type ._llcrypt registered [ 6724.796213] Key type .llcrypt registered [ 6724.838626] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6724.945296] Key type .llcrypt unregistered [ 6724.947268] Key type ._llcrypt unregistered [ 6732.035290] Key type ._llcrypt registered [ 6732.036776] Key type .llcrypt registered [ 6732.074663] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6732.441650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6732.504705] alg: No test for adler32 (adler32-zlib) [ 6733.373273] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 6733.473715] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6735.071167] Key type lgssc registered [ 6735.463725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6738.187692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6738.193929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.354947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.777485] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6743.615804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6743.727490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.710692] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6748.222603] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6749.152967] LustreError: 413177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6749.159927] LustreError: 413177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6750.176758] LustreError: 413176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.181695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6750.181740] LustreError: 413176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6750.524713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6750.649428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.937163] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6760.416202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6760.418909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.423412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6760.935380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6760.939812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.946469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6765.201865] Lustre: server umount lustre-OST0000 complete [ 6770.147184] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6770.151041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.157950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6772.713057] Lustre: server umount lustre-MDT0000 complete [ 6773.980502] LustreError: 414916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231631 with bad export cookie 470412489044053149 [ 6773.983601] LustreError: 414916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6773.983906] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6774.114656] Lustre: server umount lustre-MDT0001 complete [ 6777.656431] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6778.736766] Key type lgssc unregistered [ 6778.874510] LNet: 416322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6778.880092] LNetError: 416322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6778.890413] LNet: Removed LNI 192.168.202.116@tcp [ 6779.238114] Key type .llcrypt unregistered [ 6779.239243] Key type ._llcrypt unregistered [ 6788.523489] Key type ._llcrypt registered [ 6788.524586] Key type .llcrypt registered [ 6788.567965] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6788.917664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6788.975733] alg: No test for adler32 (adler32-zlib) [ 6789.840613] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 6789.929094] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6791.511152] Key type lgssc registered [ 6791.874211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6794.363540] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6794.369757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6809.823509] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6814.943359] LustreError: 417688:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 6841.567196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6843.531235] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6846.032112] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6848.215938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.420343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6848.445872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6849.764253] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6852.594713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6852.732212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6854.149936] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6856.554489] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6858.424635] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6862.309670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.311527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6862.312688] Lustre: Skipped 1 previous similar message [ 6866.923136] Lustre: server umount lustre-OST0000 complete [ 6868.448117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6868.450465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6868.454554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6868.456193] Lustre: Skipped 1 previous similar message [ 6872.544530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6872.545448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.353332] Lustre: server umount lustre-MDT0000 complete [ 6875.644736] LustreError: 417694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231733 with bad export cookie 6874477687326008999 [ 6875.649298] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6875.651032] LustreError: 417694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6875.786126] Lustre: server umount lustre-MDT0001 complete [ 6879.702419] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6880.732832] Key type lgssc unregistered [ 6880.863282] LNet: 421250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6880.865761] LNetError: 421250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6880.877255] LNet: Removed LNI 192.168.202.116@tcp [ 6881.195360] Key type .llcrypt unregistered [ 6881.196640] Key type ._llcrypt unregistered [ 6890.399799] Key type ._llcrypt registered [ 6890.401238] Key type .llcrypt registered [ 6890.449352] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6891.032249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6891.041813] alg: No test for adler32 (adler32-zlib) [ 6891.901499] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 6892.000839] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6893.583174] Key type lgssc registered [ 6894.115651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6896.700337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6896.706234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.865642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.208690] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6902.014660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.142054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.473211] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6905.909501] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6907.737776] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6909.949367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6910.070462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6911.990456] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6914.375952] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6922.348214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6927.842535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6927.842683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.844340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6927.852113] Lustre: Skipped 2 previous similar messages [ 6929.982642] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 6929.984541] Lustre: Skipped 2 previous similar messages [ 6931.875857] Lustre: server umount lustre-MDT0000 complete [ 6932.670785] LustreError: 422623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.16@tcp arrived at 1784231790 with bad export cookie 5556665805120012337 [ 6932.676273] LustreError: 422623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6932.962315] LustreError: 422640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.966466] LustreError: 422640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6933.331393] LustreError: 422621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784231791 with bad export cookie 5556665805120011714 [ 6933.336600] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6938.080376] LustreError: 422641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6938.080570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6938.080643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6938.081823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6938.085470] LustreError: 422641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6939.476853] Lustre: server umount lustre-OST0000 complete [ 6941.722487] Lustre: server umount lustre-MDT0001 complete [ 6945.367113] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6946.352068] Key type lgssc unregistered [ 6946.479441] LNet: 426339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6946.482122] LNetError: 426339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6946.489294] LNet: Removed LNI 192.168.202.116@tcp [ 6946.792195] Key type .llcrypt unregistered [ 6946.793191] Key type ._llcrypt unregistered [ 6955.896627] Key type ._llcrypt registered [ 6955.897722] Key type .llcrypt registered [ 6955.937088] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6956.357332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6956.387179] alg: No test for adler32 (adler32-zlib) [ 6957.238973] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 6957.333360] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6958.919097] Key type lgssc registered [ 6959.292510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6961.863865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6961.870588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.020645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6964.358102] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6967.113047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.237115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6968.578566] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6970.984840] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6972.836025] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6975.086789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6975.198910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6977.090214] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6979.447355] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6987.494352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6992.865740] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6992.867332] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6992.869746] Lustre: Skipped 1 previous similar message [ 6992.870210] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6992.871672] Lustre: Skipped 1 previous similar message [ 6995.006783] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 6997.986303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7000.121673] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 7000.125119] Lustre: Skipped 1 previous similar message [ 7005.151104] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7005.207349] Lustre: server umount lustre-MDT0001 complete [ 7005.241229] LustreError: 427724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7005.250832] LustreError: 427724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7008.224814] LustreError: 427728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.226273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7008.227195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7008.227802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7008.227806] Lustre: Skipped 2 previous similar messages [ 7012.942086] Lustre: server umount lustre-OST0000 complete [ 7013.344074] LustreError: 429159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7014.337733] Lustre: server umount lustre-MDT0000 complete [ 7018.988059] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7020.053786] Key type lgssc unregistered [ 7020.183489] LNet: 431427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7020.186534] LNetError: 431427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7020.195339] LNet: Removed LNI 192.168.202.116@tcp [ 7020.516727] Key type .llcrypt unregistered [ 7020.517870] Key type ._llcrypt unregistered [ 7028.681109] Key type ._llcrypt registered [ 7028.682438] Key type .llcrypt registered [ 7028.729976] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7034.842139] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7035.254836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7035.313050] alg: No test for adler32 (adler32-zlib) [ 7036.183600] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 7036.281955] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7037.871186] Key type lgssc registered [ 7038.234177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7050.580161] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7055.705393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7055.710607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.807384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7056.819846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7056.858465] Lustre: lustre-MDT0000: new disk, initializing [ 7056.880493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7056.886278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7058.229474] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7063.938659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.044176] Lustre: lustre-OST0000: new disk, initializing [ 7064.046684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7064.049510] Lustre: Skipped 1 previous similar message [ 7064.052506] Lustre: 436423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7064.078866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7065.954136] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7069.655304] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7069.694187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7069.696975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7069.706805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7075.092489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7075.141259] Lustre: 435405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7075.171870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7075.215306] Lustre: lustre-MDT0001: new disk, initializing [ 7075.255581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7075.268331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7075.271663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7076.617612] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7080.325373] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7080.401329] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7081.764728] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7081.836062] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7086.560669] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7086.561453] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.562778] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7086.572941] Lustre: Skipped 1 previous similar message [ 7089.491074] Lustre: server umount lustre-MDT0001 complete [ 7096.801378] LustreError: 436431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7096.806108] LustreError: 436431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7101.920210] LustreError: 435418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7101.920584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.929127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7101.930769] Lustre: Skipped 2 previous similar messages [ 7107.040417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7107.041166] LustreError: 439152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.161000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.161674] LustreError: 435417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7116.255133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7116.313075] Lustre: server umount lustre-OST0000 complete [ 7117.283106] LustreError: 439152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7123.763902] Lustre: server umount lustre-MDT0000 complete [ 7128.128187] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7129.094167] Key type lgssc unregistered [ 7129.225462] LNet: 440160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7129.228585] LNetError: 440160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7129.240506] LNet: Removed LNI 192.168.202.116@tcp [ 7129.553143] Key type .llcrypt unregistered [ 7129.554799] Key type ._llcrypt unregistered [ 7142.063396] Key type ._llcrypt registered [ 7142.064806] Key type .llcrypt registered [ 7142.105836] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7142.218324] Key type .llcrypt unregistered [ 7142.219618] Key type ._llcrypt unregistered [ 7148.971450] Key type ._llcrypt registered [ 7148.972530] Key type .llcrypt registered [ 7149.010175] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7149.314591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7149.321269] alg: No test for adler32 (adler32-zlib) [ 7150.180039] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 7150.277714] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7151.855226] Key type lgssc registered [ 7152.254325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7160.835061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7160.841275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.935807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7161.945722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7161.982229] Lustre: lustre-MDT0000: new disk, initializing [ 7162.004742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.010848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7163.347011] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7167.231328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.272907] Lustre: 443502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7167.293660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7167.295916] Lustre: Skipped 1 previous similar message [ 7167.329933] Lustre: lustre-MDT0001: new disk, initializing [ 7167.358951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7167.369629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7167.372364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7168.592818] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7172.218336] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7174.172876] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7176.939576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7177.137629] Lustre: lustre-OST0000: new disk, initializing [ 7177.139105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7177.141873] Lustre: 445465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7177.181735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7180.272798] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7183.365356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7183.376708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7183.440163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7184.923764] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7188.450883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.454420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7188.456882] Lustre: Skipped 2 previous similar messages [ 7188.464845] Lustre: Skipped 2 previous similar messages [ 7193.572383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7193.575381] Lustre: Skipped 2 previous similar messages [ 7196.369368] Lustre: server umount lustre-MDT0000 complete [ 7199.191994] LustreError: 443495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784232057 with bad export cookie 1912258934870488428 [ 7199.198463] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7199.202533] LustreError: 443495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7199.485811] Lustre: server umount lustre-MDT0001 complete [ 7213.221044] Lustre: server umount lustre-OST0000 complete [ 7219.746538] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:01:17 (1784232077) [ 7224.824854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.149110] LustreError: 447498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7225.157172] LustreError: 447498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7225.206414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7228.263890] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7230.453455] LustreError: 447498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7233.662267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7233.909872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7236.847145] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7241.003909] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7244.397467] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7260.258585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7260.335490] Lustre: Found index 0 for lustre-OST0000, updating log [ 7260.420768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7262.757200] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7265.868240] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7271.733570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7279.073688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7279.080148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7284.179172] Lustre: server umount lustre-OST0000 complete [ 7300.575937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7300.580048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.587119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7300.589182] Lustre: Skipped 2 previous similar messages [ 7306.210416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7306.213922] Lustre: Skipped 3 previous similar messages [ 7311.327175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7311.389159] Lustre: server umount lustre-MDT0000 complete [ 7312.351926] LustreError: 447938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.357393] LustreError: 447938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7312.785290] LustreError: 447480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784232170 with bad export cookie 1912258934870489807 [ 7312.788153] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7312.789273] LustreError: 447480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7312.908326] Lustre: server umount lustre-MDT0001 complete [ 7316.175141] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:02:54 (1784232174) [ 7324.385850] LNet: 1 local NIs in recovery (showing 1): 192.168.202.116@tcp [ 7907.520114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7907.908616] LustreError: 453827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7908.000193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7911.869653] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7913.452554] LustreError: 453828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.552291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.739627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7921.357952] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7926.692329] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7930.635042] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7948.393816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7948.606288] Lustre: Found index 0 for lustre-OST0000, updating log [ 7948.763415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7952.984624] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7955.956769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7958.097013] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7961.057556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7961.061728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7961.068186] Lustre: Skipped 1 previous similar message [ 7961.085104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7961.087948] Lustre: Skipped 2 previous similar messages [ 7964.383426] LNet: 1 local NIs in recovery (showing 1): 192.168.202.116@tcp [ 7965.758455] Lustre: server umount lustre-OST0000 complete [ 7969.632758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7969.635380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7969.640709] Lustre: Skipped 1 previous similar message [ 7971.683908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7971.692294] Lustre: Skipped 4 previous similar messages [ 7974.760369] LustreError: 453823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7974.781277] LustreError: 453823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7974.804950] Lustre: server umount lustre-MDT0000 complete [ 7977.425661] LustreError: 453808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784232835 with bad export cookie 1912258934870490878 [ 7977.432470] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7977.439506] LustreError: 453808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7977.750597] Lustre: server umount lustre-MDT0001 complete [ 8001.397361] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8008.335244] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8035.051727] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:14:52 (1784232892) [ 8048.464873] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8050.909733] Key type lgssc unregistered [ 8051.164678] LNet: 460429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8051.173304] LNetError: 460429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8168.934606] LNet: Removed LNI 192.168.202.116@tcp [ 8169.427539] Key type .llcrypt unregistered [ 8169.428792] Key type ._llcrypt unregistered [ 8181.141098] Key type ._llcrypt registered [ 8181.142980] Key type .llcrypt registered [ 8181.216987] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8182.051852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8182.131610] alg: No test for adler32 (adler32-zlib) [ 8183.119474] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 8183.279161] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8184.895225] Key type lgssc registered [ 8185.648580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8188.350489] LNet: Added LNI 192.168.202.116@tcp100 [8/256/0/180] [ 8188.927514] LNet: Added LNI 192.168.202.116@tcp101 [8/256/0/180] [ 8189.594637] LNet: Added LNI 192.168.202.116@tcp102 [8/256/0/180] [ 8190.182322] LNet: Added LNI 192.168.202.116@tcp103 [8/256/0/180] [ 8190.791976] LNet: Added LNI 192.168.202.116@tcp104 [8/256/0/180] [ 8191.439291] LNet: Added LNI 192.168.202.116@tcp105 [8/256/0/180] [ 8191.944095] LNet: Added LNI 192.168.202.116@tcp106 [8/256/0/180] [ 8192.415893] LNet: Added LNI 192.168.202.116@tcp107 [8/256/0/180] [ 8192.963340] LNet: Added LNI 192.168.202.116@tcp108 [8/256/0/180] [ 8193.514798] LNet: Added LNI 192.168.202.116@tcp109 [8/256/0/180] [ 8194.049359] LNet: Added LNI 192.168.202.116@tcp110 [8/256/0/180] [ 8194.621566] LNet: Added LNI 192.168.202.116@tcp111 [8/256/0/180] [ 8195.115513] LNet: Added LNI 192.168.202.116@tcp112 [8/256/0/180] [ 8195.795182] LNet: Added LNI 192.168.202.116@tcp113 [8/256/0/180] [ 8196.350255] LNet: Added LNI 192.168.202.116@tcp114 [8/256/0/180] [ 8196.861156] LNet: Added LNI 192.168.202.116@tcp115 [8/256/0/180] [ 8197.388876] LNet: Added LNI 192.168.202.116@tcp116 [8/256/0/180] [ 8197.879788] LNet: Added LNI 192.168.202.116@tcp117 [8/256/0/180] [ 8198.400617] LNet: Added LNI 192.168.202.116@tcp118 [8/256/0/180] [ 8198.931913] LNet: Added LNI 192.168.202.116@tcp119 [8/256/0/180] [ 8199.531495] LNet: Added LNI 192.168.202.116@tcp120 [8/256/0/180] [ 8200.073668] LNet: Added LNI 192.168.202.116@tcp121 [8/256/0/180] [ 8200.641989] LNet: Added LNI 192.168.202.116@tcp122 [8/256/0/180] [ 8201.212630] LNet: Added LNI 192.168.202.116@tcp123 [8/256/0/180] [ 8201.751282] LNet: Added LNI 192.168.202.116@tcp124 [8/256/0/180] [ 8202.288954] LNet: Added LNI 192.168.202.116@tcp125 [8/256/0/180] [ 8202.778546] LNet: Added LNI 192.168.202.116@tcp126 [8/256/0/180] [ 8203.389287] LNet: Added LNI 192.168.202.116@tcp127 [8/256/0/180] [ 8203.945600] LNet: Added LNI 192.168.202.116@tcp128 [8/256/0/180] [ 8204.520520] LNet: Added LNI 192.168.202.116@tcp129 [8/256/0/180] [ 8205.088467] LNet: Added LNI 192.168.202.116@tcp130 [8/256/0/180] [ 8205.653920] LNet: Added LNI 192.168.202.116@tcp131 [8/256/0/180] [ 8206.307796] LNet: Added LNI 192.168.202.116@tcp132 [8/256/0/180] [ 8206.886689] LNet: Added LNI 192.168.202.116@tcp133 [8/256/0/180] [ 8207.516301] LNet: Added LNI 192.168.202.116@tcp134 [8/256/0/180] [ 8208.129772] LNet: Added LNI 192.168.202.116@tcp135 [8/256/0/180] [ 8212.580964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8212.597495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8213.661463] Lustre: 463415:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.116@tcp: can't add NID 192.168.202.116@tcp130: rc = -75 [ 8213.729507] LNetError: 463438:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.116 [ 8213.744718] LNetError: Protocol error connecting to 192.168.202.116@tcp125 at host 192.168.202.116:988: is it running a compatible version of Lustre? [ 8213.752469] LNetError: Skipped 7 previous similar messages [ 8213.808811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8213.848433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8213.896329] Lustre: lustre-MDT0000: new disk, initializing [ 8213.953671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8213.965418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8213.984065] LNet: 3 local NIs in recovery (showing 3): 192.168.202.116@tcp127, 192.168.202.116@tcp128, 192.168.202.116@tcp125 [ 8214.752778] LNetError: 463442:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.116 [ 8214.771131] LNetError: 463442:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 8214.778577] LNetError: Protocol error connecting to 192.168.202.116@tcp104 at host 192.168.202.116:988: is it running a compatible version of Lustre? [ 8217.056505] LNetError: 463435:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.116 [ 8217.061912] LNetError: 463435:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 8217.066426] LNetError: Protocol error connecting to 192.168.202.116@tcp127 at host 192.168.202.116:988: is it running a compatible version of Lustre? [ 8217.080363] LNetError: Skipped 3 previous similar messages [ 8217.388323] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8221.154765] LNetError: 463438:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.116 [ 8221.168911] LNetError: Protocol error connecting to 192.168.202.116@tcp127 at host 192.168.202.116:988: is it running a compatible version of Lustre? [ 8224.773394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8224.878893] Lustre: 463431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8224.925941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8224.930735] Lustre: Skipped 1 previous similar message [ 8224.991601] Lustre: lustre-MDT0001: new disk, initializing [ 8225.013459] Lustre: 464232:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.116@tcp: can't add NID 192.168.202.116@tcp130: rc = -75 [ 8225.017888] Lustre: 464232:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8225.108198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8225.155508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8225.161474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8226.865250] Lustre: 464374:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.116@tcp: can't add NID 192.168.202.116@tcp130: rc = -75 [ 8226.884713] Lustre: 464374:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8228.174315] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8229.344850] LNetError: 463448:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.116 [ 8229.350788] LNetError: Protocol error connecting to 192.168.202.116@tcp127 at host 192.168.202.116:988: is it running a compatible version of Lustre? [ 8234.860818] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8239.145109] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8243.681963] LNetError: 463443:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.116 [ 8243.689755] LNetError: Protocol error connecting to 192.168.202.116@tcp120 at host 192.168.202.116:988: is it running a compatible version of Lustre? [ 8244.446303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8244.659429] Lustre: lustre-OST0000: new disk, initializing [ 8244.661843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8244.664288] Lustre: 465403:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8244.716737] Lustre: 465408:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.116@tcp: can't add NID 192.168.202.116@tcp130: rc = -75 [ 8244.719927] Lustre: 465408:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 8244.744972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.921645] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8251.402954] Lustre: 465877:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.116@tcp: can't add NID 192.168.202.116@tcp130: rc = -75 [ 8251.417466] Lustre: 465877:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8251.463982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8251.469572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8251.583263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8255.698981] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8260.068731] LNetError: 463446:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.116 [ 8260.079307] LNetError: 463446:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [ 8260.092695] LNetError: Protocol error connecting to 192.168.202.116@tcp126 at host 192.168.202.116:988: is it running a compatible version of Lustre? [ 8260.103110] LNetError: Skipped 11 previous similar messages [ 8262.112217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8262.116473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8262.118314] LustreError: Skipped 1 previous similar message [ 8262.125385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8262.135398] Lustre: Skipped 1 previous similar message [ 8265.187435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8265.190519] Lustre: Skipped 1 previous similar message [ 8267.312436] Lustre: server umount lustre-OST0000 complete [ 8270.310917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.322771] Lustre: Skipped 1 previous similar message [ 8270.328639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8270.332384] Lustre: Skipped 1 previous similar message [ 8273.376465] LNet: 1 local NIs in recovery (showing 1): 192.168.202.116@tcp127 [ 8275.425062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8275.429050] Lustre: Skipped 1 previous similar message [ 8276.373469] Lustre: server umount lustre-MDT0000 complete [ 8279.259815] LustreError: 463425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233137 with bad export cookie 13409860129156774238 [ 8279.263228] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8279.267446] LustreError: 463425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8279.607989] Lustre: server umount lustre-MDT0001 complete [ 8287.103625] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8289.278430] Key type lgssc unregistered [ 8289.641557] LNet: 467308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8289.654502] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8290.719435] LNet: Removed LNI 192.168.202.116@tcp [ 8290.723222] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8290.733745] LNet: Removed LNI 192.168.202.116@tcp100 [ 8291.743285] LNet: Removed LNI 192.168.202.116@tcp101 [ 8291.746659] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8291.751596] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8291.756317] LNet: Removed LNI 192.168.202.116@tcp102 [ 8292.767239] LNet: Removed LNI 192.168.202.116@tcp103 [ 8293.791207] LNet: Removed LNI 192.168.202.116@tcp104 [ 8293.795716] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8293.799888] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8293.812704] LNet: Removed LNI 192.168.202.116@tcp105 [ 8294.879670] LNet: Removed LNI 192.168.202.116@tcp106 [ 8295.905102] LNet: Removed LNI 192.168.202.116@tcp107 [ 8296.927514] LNet: Removed LNI 192.168.202.116@tcp108 [ 8297.951162] LNet: Removed LNI 192.168.202.116@tcp109 [ 8297.954518] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8297.959482] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8298.975172] LNet: Removed LNI 192.168.202.116@tcp110 [ 8299.999130] LNet: Removed LNI 192.168.202.116@tcp111 [ 8301.023202] LNet: Removed LNI 192.168.202.116@tcp112 [ 8302.047256] LNet: Removed LNI 192.168.202.116@tcp113 [ 8303.071226] LNet: Removed LNI 192.168.202.116@tcp114 [ 8304.095420] LNet: Removed LNI 192.168.202.116@tcp115 [ 8305.119531] LNet: Removed LNI 192.168.202.116@tcp116 [ 8306.143183] LNet: Removed LNI 192.168.202.116@tcp117 [ 8306.147149] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8306.155229] LNetError: 467308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 8306.160509] LNet: Removed LNI 192.168.202.116@tcp118 [ 8307.169223] LNet: Removed LNI 192.168.202.116@tcp119 [ 8308.191255] LNet: Removed LNI 192.168.202.116@tcp120 [ 8309.215657] LNet: Removed LNI 192.168.202.116@tcp121 [ 8310.239145] LNet: Removed LNI 192.168.202.116@tcp122 [ 8311.263162] LNet: Removed LNI 192.168.202.116@tcp123 [ 8312.287307] LNet: Removed LNI 192.168.202.116@tcp124 [ 8313.311240] LNet: Removed LNI 192.168.202.116@tcp125 [ 8314.335165] LNet: Removed LNI 192.168.202.116@tcp126 [ 8314.338925] LNet: Removed LNI 192.168.202.116@tcp127 [ 8315.359145] LNet: Removed LNI 192.168.202.116@tcp128 [ 8316.383377] LNet: Removed LNI 192.168.202.116@tcp129 [ 8316.387498] LNet: Removed LNI 192.168.202.116@tcp130 [ 8316.392947] LNet: Removed LNI 192.168.202.116@tcp131 [ 8316.399885] LNet: Removed LNI 192.168.202.116@tcp132 [ 8316.402821] LNet: Removed LNI 192.168.202.116@tcp133 [ 8316.406202] LNet: Removed LNI 192.168.202.116@tcp134 [ 8316.416548] LNet: Removed LNI 192.168.202.116@tcp135 [ 8316.937713] Key type .llcrypt unregistered [ 8316.939609] Key type ._llcrypt unregistered [ 8332.809987] Key type ._llcrypt registered [ 8332.812498] Key type .llcrypt registered [ 8332.904312] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8342.241250] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8342.858186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8342.969684] alg: No test for adler32 (adler32-zlib) [ 8343.950834] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 8344.099670] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8345.751163] Key type lgssc registered [ 8346.534619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8371.273657] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:20:28 (1784233228) [ 8382.591508] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8384.231744] Key type lgssc unregistered [ 8384.448093] LNet: 471358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8384.452271] LNetError: 471358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8384.468127] LNet: Removed LNI 192.168.202.116@tcp [ 8385.016399] Key type .llcrypt unregistered [ 8385.017944] Key type ._llcrypt unregistered [ 8395.895371] Key type ._llcrypt registered [ 8395.896526] Key type .llcrypt registered [ 8395.950216] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8396.524090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8396.552733] alg: No test for adler32 (adler32-zlib) [ 8397.513599] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 8397.675962] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8399.303280] Key type lgssc registered [ 8400.354583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8410.630855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8410.743502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.238943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8412.294083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8412.409558] Lustre: lustre-MDT0000: new disk, initializing [ 8412.813799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.902523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8420.231758] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8435.921751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8436.084354] Lustre: 472544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8436.221133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8436.226421] Lustre: Skipped 1 previous similar message [ 8436.370501] Lustre: lustre-MDT0001: new disk, initializing [ 8436.538051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8436.577753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8436.607238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8442.632959] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8452.967769] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8459.519859] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8466.967231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8467.260956] Lustre: lustre-OST0000: new disk, initializing [ 8467.265281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8467.270696] Lustre: 474505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8467.369091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8473.910728] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8477.269461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8477.276609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8477.339828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8483.459661] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8488.404812] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing lnet_if_list [ 8493.584115] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8493.605830] LNet: Added LNI 192.168.202.116@tcp10 [8/256/0/180] [ 8498.654957] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8498.675336] LNet: Added LNI 192.168.202.116@tcp11 [8/256/0/180] [ 8503.331716] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8503.379453] LNet: Added LNI 192.168.202.116@tcp12 [8/256/0/180] [ 8508.367363] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8508.398377] LNet: Added LNI 192.168.202.116@tcp13 [8/256/0/180] [ 8512.534209] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8512.554472] LNet: Added LNI 192.168.202.116@tcp14 [8/256/0/180] [ 8516.486364] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8516.505266] LNet: Added LNI 192.168.202.116@tcp15 [8/256/0/180] [ 8519.740305] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8519.761209] LNet: Added LNI 192.168.202.116@tcp16 [8/256/0/180] [ 8524.215424] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8524.236188] LNet: Added LNI 192.168.202.116@tcp17 [8/256/0/180] [ 8528.267769] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8528.287144] LNet: Added LNI 192.168.202.116@tcp18 [8/256/0/180] [ 8532.947868] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8532.973811] LNet: Added LNI 192.168.202.116@tcp19 [8/256/0/180] [ 8538.077489] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8538.097961] LNet: Added LNI 192.168.202.116@tcp20 [8/256/0/180] [ 8542.966947] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8542.998857] LNet: Added LNI 192.168.202.116@tcp21 [8/256/0/180] [ 8547.865572] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8547.897220] LNet: Added LNI 192.168.202.116@tcp22 [8/256/0/180] [ 8552.554915] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8552.576283] LNet: Added LNI 192.168.202.116@tcp23 [8/256/0/180] [ 8557.319730] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8557.345227] LNet: Added LNI 192.168.202.116@tcp24 [8/256/0/180] [ 8561.587893] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8561.613683] LNet: Added LNI 192.168.202.116@tcp25 [8/256/0/180] [ 8566.322247] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8566.344717] LNet: Added LNI 192.168.202.116@tcp26 [8/256/0/180] [ 8570.824690] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8570.854423] LNet: Added LNI 192.168.202.116@tcp27 [8/256/0/180] [ 8575.273942] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8575.296747] LNet: Added LNI 192.168.202.116@tcp28 [8/256/0/180] [ 8579.810587] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8579.832194] LNet: Added LNI 192.168.202.116@tcp29 [8/256/0/180] [ 8584.628319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8584.665660] LNet: Added LNI 192.168.202.116@tcp30 [8/256/0/180] [ 8589.739898] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8589.764968] LNet: Added LNI 192.168.202.116@tcp31 [8/256/0/180] [ 8595.254995] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8595.281279] LNet: Added LNI 192.168.202.116@tcp32 [8/256/0/180] [ 8601.400882] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8601.427970] LNet: Added LNI 192.168.202.116@tcp33 [8/256/0/180] [ 8606.937777] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8606.964662] LNet: Added LNI 192.168.202.116@tcp34 [8/256/0/180] [ 8612.392423] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8612.430807] LNet: Added LNI 192.168.202.116@tcp35 [8/256/0/180] [ 8617.820665] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8617.852473] LNet: Added LNI 192.168.202.116@tcp36 [8/256/0/180] [ 8623.169496] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8623.201476] LNet: Added LNI 192.168.202.116@tcp37 [8/256/0/180] [ 8628.968712] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8629.006657] LNet: Added LNI 192.168.202.116@tcp38 [8/256/0/180] [ 8634.587282] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8634.620583] LNet: Added LNI 192.168.202.116@tcp39 [8/256/0/180] [ 8641.523324] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8641.543814] LNet: Added LNI 192.168.202.116@tcp40 [8/256/0/180] [ 8648.169352] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8648.200821] LNet: Added LNI 192.168.202.116@tcp41 [8/256/0/180] [ 8654.228873] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8654.261505] LNet: Added LNI 192.168.202.116@tcp42 [8/256/0/180] [ 8661.261544] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8661.283994] LNet: Added LNI 192.168.202.116@tcp43 [8/256/0/180] [ 8668.635919] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8668.659139] LNet: Added LNI 192.168.202.116@tcp44 [8/256/0/180] [ 8676.203682] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8676.233946] LNet: Added LNI 192.168.202.116@tcp45 [8/256/0/180] [ 8683.382033] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8683.408772] LNet: Added LNI 192.168.202.116@tcp46 [8/256/0/180] [ 8691.141774] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8691.175523] LNet: Added LNI 192.168.202.116@tcp47 [8/256/0/180] [ 8698.758316] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8698.782554] LNet: Added LNI 192.168.202.116@tcp48 [8/256/0/180] [ 8705.697753] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8705.722670] LNet: Added LNI 192.168.202.116@tcp49 [8/256/0/180] [ 8714.616398] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8714.647927] LNet: Added LNI 192.168.202.116@tcp50 [8/256/0/180] [ 8733.667745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8733.708790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8737.883425] Lustre: server umount lustre-MDT0000 complete [ 8741.695889] LustreError: 472538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233599 with bad export cookie 9205838079209067014 [ 8741.698929] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8741.706290] LustreError: 472538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8741.978675] Lustre: server umount lustre-MDT0001 complete [ 8755.016545] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8759.265270] Lustre: 471822:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233601/real 1784233601] req@000000004a5bd9c1 x1870904169743360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8759.278584] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8759.283770] Lustre: Skipped 2 previous similar messages [ 8760.153873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.080514] Lustre: 471821:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233604/real 1784233604] req@00000000acf563d5 x1870904169743488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8764.383148] Lustre: 471819:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233606/real 1784233606] req@0000000036271d9c x1870904169743616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8766.434479] LustreError: 471818:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000036271d9c x1870904169744384/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8766.642200] LustreError: 474498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.651623] LustreError: 474498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8766.667176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8767.455196] Lustre: 471820:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233609/real 1784233609] req@000000007adabd68 x1870904169743872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8768.674503] LustreError: 474499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8769.710168] LustreError: 474498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8769.734150] LustreError: 474498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8770.001656] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8773.088235] Lustre: 471822:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233614/real 1784233614] req@0000000049068cd0 x1870904169744000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8775.138888] LustreError: 498677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8775.151865] LustreError: 498677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8775.863095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8776.067624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8778.076471] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8781.290756] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8782.735417] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8786.542881] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8788.454388] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8798.689339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8798.691910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8798.695910] Lustre: Skipped 1 previous similar message [ 8798.699410] Lustre: Skipped 2 previous similar messages [ 8800.689452] Lustre: server umount lustre-OST0000 complete [ 8803.809856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8803.816818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8803.821572] Lustre: Skipped 1 previous similar message [ 8803.829434] Lustre: Skipped 2 previous similar messages [ 8808.929742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8808.935552] Lustre: Skipped 1 previous similar message [ 8809.158747] Lustre: server umount lustre-MDT0000 complete [ 8811.417387] LustreError: 472536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233669 with bad export cookie 9205838079209069450 [ 8811.421316] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8811.422539] LustreError: 472536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8811.558602] Lustre: server umount lustre-MDT0001 complete [ 8817.095751] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8818.575959] Key type lgssc unregistered [ 8818.791305] LNet: 501446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8818.796361] LNetError: 501446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8818.800570] LNet: Removed LNI 192.168.202.116@tcp [ 8818.804816] LNet: Removed LNI 192.168.202.116@tcp10 [ 8818.806822] LNet: Removed LNI 192.168.202.116@tcp11 [ 8818.808700] LNet: Removed LNI 192.168.202.116@tcp12 [ 8818.810480] LNet: Removed LNI 192.168.202.116@tcp13 [ 8818.811737] LNet: Removed LNI 192.168.202.116@tcp14 [ 8818.813614] LNet: Removed LNI 192.168.202.116@tcp15 [ 8818.815364] LNet: Removed LNI 192.168.202.116@tcp16 [ 8818.817059] LNet: Removed LNI 192.168.202.116@tcp17 [ 8818.819068] LNet: Removed LNI 192.168.202.116@tcp18 [ 8818.820637] LNet: Removed LNI 192.168.202.116@tcp19 [ 8818.822120] LNet: Removed LNI 192.168.202.116@tcp20 [ 8818.823595] LNet: Removed LNI 192.168.202.116@tcp21 [ 8818.825784] LNet: Removed LNI 192.168.202.116@tcp22 [ 8818.827465] LNet: Removed LNI 192.168.202.116@tcp23 [ 8818.828626] LNet: Removed LNI 192.168.202.116@tcp24 [ 8818.830734] LNet: Removed LNI 192.168.202.116@tcp25 [ 8818.832238] LNet: Removed LNI 192.168.202.116@tcp26 [ 8818.834390] LNet: Removed LNI 192.168.202.116@tcp27 [ 8818.836029] LNet: Removed LNI 192.168.202.116@tcp28 [ 8818.839278] LNet: Removed LNI 192.168.202.116@tcp29 [ 8818.841042] LNet: Removed LNI 192.168.202.116@tcp30 [ 8818.843014] LNet: Removed LNI 192.168.202.116@tcp31 [ 8818.844361] LNet: Removed LNI 192.168.202.116@tcp32 [ 8818.846069] LNet: Removed LNI 192.168.202.116@tcp33 [ 8818.848055] LNet: Removed LNI 192.168.202.116@tcp34 [ 8818.849702] LNet: Removed LNI 192.168.202.116@tcp35 [ 8818.851173] LNet: Removed LNI 192.168.202.116@tcp36 [ 8818.852975] LNet: Removed LNI 192.168.202.116@tcp37 [ 8818.854772] LNet: Removed LNI 192.168.202.116@tcp38 [ 8818.856592] LNet: Removed LNI 192.168.202.116@tcp39 [ 8818.858243] LNet: Removed LNI 192.168.202.116@tcp40 [ 8818.859586] LNet: Removed LNI 192.168.202.116@tcp41 [ 8818.861062] LNet: Removed LNI 192.168.202.116@tcp42 [ 8818.862445] LNet: Removed LNI 192.168.202.116@tcp43 [ 8818.863951] LNet: Removed LNI 192.168.202.116@tcp44 [ 8818.865684] LNet: Removed LNI 192.168.202.116@tcp45 [ 8818.867444] LNet: Removed LNI 192.168.202.116@tcp46 [ 8818.868783] LNet: Removed LNI 192.168.202.116@tcp47 [ 8818.870234] LNet: Removed LNI 192.168.202.116@tcp48 [ 8818.871788] LNet: Removed LNI 192.168.202.116@tcp49 [ 8818.879671] LNet: Removed LNI 192.168.202.116@tcp50 [ 8819.262517] Key type .llcrypt unregistered [ 8819.263589] Key type ._llcrypt unregistered [ 8829.397789] Key type ._llcrypt registered [ 8829.399370] Key type .llcrypt registered [ 8829.454217] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8838.821349] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8839.374133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8839.411586] alg: No test for adler32 (adler32-zlib) [ 8840.374576] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 8840.530480] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8842.151158] Key type lgssc registered [ 8842.827679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8862.874865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8862.894464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.022033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8864.036304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8864.084349] Lustre: lustre-MDT0000: new disk, initializing [ 8864.123152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8864.137060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8866.770876] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8873.423437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8873.502726] Lustre: 504612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8873.538630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8873.540669] Lustre: Skipped 1 previous similar message [ 8873.593061] Lustre: lustre-MDT0001: new disk, initializing [ 8873.639197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8873.662463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8873.672994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8875.959587] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8881.287601] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8883.923570] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8888.293494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8888.400427] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8888.446166] Lustre: lustre-OST0000: new disk, initializing [ 8888.450269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8888.454493] Lustre: 506625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8888.503974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8890.758721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8890.762964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8890.835307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8891.896552] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8897.735400] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8906.208678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8906.209580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8906.213905] LustreError: Skipped 1 previous similar message [ 8906.225940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8906.232589] Lustre: Skipped 1 previous similar message [ 8908.087833] Lustre: server umount lustre-OST0000 complete [ 8911.329652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8911.335413] Lustre: Skipped 1 previous similar message [ 8911.343893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8911.347658] Lustre: Skipped 1 previous similar message [ 8916.083216] Lustre: server umount lustre-MDT0000 complete [ 8918.249749] LustreError: 505406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233776 with bad export cookie 14860761120629154487 [ 8918.256123] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8918.452581] Lustre: server umount lustre-MDT0001 complete [ 8923.961377] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8925.321197] Key type lgssc unregistered [ 8925.492536] LNet: 508522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8925.496830] LNetError: 508522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8925.507687] LNet: Removed LNI 192.168.202.116@tcp [ 8925.939668] Key type .llcrypt unregistered [ 8925.940986] Key type ._llcrypt unregistered [ 8936.815843] Key type ._llcrypt registered [ 8936.817935] Key type .llcrypt registered [ 8936.871652] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8937.480756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8937.491429] alg: No test for adler32 (adler32-zlib) [ 8938.463132] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 8938.614625] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8940.231188] Key type lgssc registered [ 8940.691712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8944.287412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8944.300679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8945.532188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8947.345359] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8951.203905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.373120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8953.160700] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8956.379791] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8959.375595] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8962.791770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8962.966982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8965.277955] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8968.465201] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8983.140187] loop0: detected capacity change from 0 to 204800000 [ 8983.220716] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8983.225450] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8983.232680] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8985.900697] loop0: detected capacity change from 0 to 204800000 [ 8985.967664] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8985.972640] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8985.977925] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8988.444701] loop0: detected capacity change from 0 to 204800000 [ 8990.918564] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8990.986056] Lustre: fs15246-OST0000: new disk, initializing [ 8990.989255] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8990.994876] Lustre: 513371:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8991.023927] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8993.527830] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8998.492512] loop1: detected capacity change from 0 to 204800000 [ 9001.317426] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9001.384439] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9001.425049] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9001.486901] Lustre: fs15246-MDT0000: new disk, initializing [ 9001.568429] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.593453] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9001.646871] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9002.659429] Lustre: 514490:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001948a6f6 x1870904736910592/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784233876 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9003.530703] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9007.715985] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9007.720634] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9007.733848] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9017.317437] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.322784] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9021.151839] Lustre: server umount fs15246-MDT0000 complete [ 9022.437884] LustreError: 512208:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9022.453096] LustreError: 512208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9025.378840] Lustre: server umount fs15246-OST0000 complete [ 9031.648795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9031.651820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9031.657693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9032.674036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9032.676308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9032.680513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9036.955849] Lustre: server umount lustre-OST0000 complete [ 9042.912508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9042.924651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9042.928724] Lustre: Skipped 2 previous similar messages [ 9048.033387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9048.037958] Lustre: Skipped 1 previous similar message [ 9053.154588] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9053.229300] Lustre: server umount lustre-MDT0000 complete [ 9055.038910] LustreError: 510499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233913 with bad export cookie 1639224373476876380 [ 9055.043850] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9055.044050] LustreError: 510499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9055.198210] Lustre: server umount lustre-MDT0001 complete [ 9060.413287] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9062.049672] Key type lgssc unregistered [ 9062.197662] LNet: 517574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9062.200939] LNetError: 517574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9062.207541] LNet: Removed LNI 192.168.202.116@tcp [ 9062.632607] Key type .llcrypt unregistered [ 9062.633991] Key type ._llcrypt unregistered [ 9073.356314] Key type ._llcrypt registered [ 9073.357914] Key type .llcrypt registered [ 9073.418381] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9073.864614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9073.888394] alg: No test for adler32 (adler32-zlib) [ 9074.902792] Lustre: Lustre: Build Version: 2.17.54_162_gb497eb5 [ 9075.071437] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 9076.703214] Key type lgssc registered [ 9077.334578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9096.635445] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9101.717754] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9120.030230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9120.043534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9121.192405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9121.207854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9121.258990] Lustre: lustre-MDT0000: new disk, initializing [ 9121.298921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9121.312958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9123.372814] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9128.944926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9129.001718] Lustre: 522004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9129.043988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9129.047235] Lustre: Skipped 1 previous similar message [ 9129.090487] Lustre: lustre-MDT0001: new disk, initializing [ 9129.160233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9129.177395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9129.183070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9131.436223] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9136.401937] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9139.272979] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9142.789982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9142.980353] Lustre: lustre-OST0000: new disk, initializing [ 9142.983503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9142.987226] Lustre: 523964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9143.025520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9145.631435] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9148.431984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9148.443032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9148.490109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9150.507103] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9152.930935] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9153.054865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9155.258207] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9155.363253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9158.631990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9158.632907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9158.634233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9158.635210] LustreError: Skipped 1 previous similar message [ 9158.643930] Lustre: Skipped 1 previous similar message [ 9162.594122] Lustre: server umount lustre-OST0000 complete [ 9165.280591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9165.284985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9165.292743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9165.297873] Lustre: Skipped 1 previous similar message [ 9168.865491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9168.865640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9170.700785] Lustre: server umount lustre-MDT0000 complete [ 9172.627631] LustreError: 522795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234030 with bad export cookie 2529029122370434878 [ 9172.629768] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9172.635550] LustreError: 522795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9172.820907] Lustre: server umount lustre-MDT0001 complete [ 9186.366304] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9192.037658] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9208.865595] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8991 sec ========== 16:34:26 (1784234066) [ 9209.879734] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:34:27 (1784234067) === [ 9211.555995] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:34:29 (1784234069) === [ 9232.476777] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9233.930347] Key type lgssc unregistered [ 9234.106693] LNet: 530587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9234.110229] LNetError: 530587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9234.119311] LNet: Removed LNI 192.168.202.116@tcp [ 9234.545306] Key type .llcrypt unregistered [ 9234.547471] Key type ._llcrypt unregistered