[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461530118 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003243] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.021215] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025320] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] *** VALIDATE tmpfs *** [ 0.031408] *** VALIDATE proc *** [ 0.032246] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034226] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037028] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041362] debug: unmapping init [mem 0xffffffff87c59000-0xffffffff87c60fff] [ 0.043700] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044688] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051024] ... event mask: 000000070000000f [ 0.052295] rcu: Hierarchical SRCU implementation. [ 0.054374] smp: Bringing up secondary CPUs ... [ 0.055560] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.060011] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129023] node 0 deferred pages initialised in 64ms [ 0.132229] devtmpfs: initialized [ 0.133232] x86/mm: Memory block size: 128MB [ 0.135540] gcov: version magic: 0x41383552 [ 0.138056] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145269] pinctrl core: initialized pinctrl subsystem [ 0.148253] [ 0.148838] ************************************************************* [ 0.151015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154012] ** ** [ 0.156011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159014] ** ** [ 0.161010] ** This means that this kernel is built to expose internal ** [ 0.164013] ** IOMMU data structures, which may compromise security on ** [ 0.167018] ** your system. ** [ 0.169011] ** ** [ 0.171014] ** If you see this message and you are not debugging the ** [ 0.174012] ** kernel, report this immediately to your vendor! ** [ 0.176015] ** ** [ 0.179013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181044] ************************************************************* [ 0.184713] NET: Registered protocol family 16 [ 0.186323] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194104] cpuidle: using governor menu [ 0.195690] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196423] PCI: Using configuration type 1 for base access [ 0.197171] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207155] cryptd: max_cpu_qlen set to 1000 [ 0.208242] ACPI: Added _OSI(Module Device) [ 0.210086] ACPI: Added _OSI(Processor Device) [ 0.212012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218646] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225361] ACPI: Interpreter enabled [ 0.227060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228012] ACPI: Using IOAPIC for interrupt routing [ 0.230154] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.244736] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.257701] acpiphp: Slot [2] registered [ 0.259113] acpiphp: Slot [5] registered [ 0.261107] acpiphp: Slot [6] registered [ 0.262130] acpiphp: Slot [7] registered [ 0.264091] acpiphp: Slot [8] registered [ 0.265095] acpiphp: Slot [9] registered [ 0.266154] acpiphp: Slot [10] registered [ 0.268137] acpiphp: Slot [3] registered [ 0.269096] acpiphp: Slot [4] registered [ 0.271108] acpiphp: Slot [11] registered [ 0.273102] acpiphp: Slot [12] registered [ 0.274146] acpiphp: Slot [13] registered [ 0.276110] acpiphp: Slot [14] registered [ 0.278101] acpiphp: Slot [15] registered [ 0.279184] acpiphp: Slot [16] registered [ 0.281108] acpiphp: Slot [17] registered [ 0.283111] acpiphp: Slot [18] registered [ 0.285125] acpiphp: Slot [19] registered [ 0.286101] acpiphp: Slot [20] registered [ 0.288107] acpiphp: Slot [21] registered [ 0.289108] acpiphp: Slot [22] registered [ 0.291096] acpiphp: Slot [23] registered [ 0.292095] acpiphp: Slot [24] registered [ 0.294075] acpiphp: Slot [25] registered [ 0.295080] acpiphp: Slot [26] registered [ 0.296105] acpiphp: Slot [27] registered [ 0.298181] acpiphp: Slot [28] registered [ 0.300129] acpiphp: Slot [29] registered [ 0.301108] acpiphp: Slot [30] registered [ 0.303088] acpiphp: Slot [31] registered [ 0.304064] PCI host bridge to bus 0000:00 [ 0.305019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310054] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312112] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.318025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.320187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.324032] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327374] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336829] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.342056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.347016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.352571] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354739] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360866] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.366014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.379015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.390895] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.405015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.429494] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.437015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.446016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.470000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.475015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.481014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.504019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.514013] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.520014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.527017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.543013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.553987] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.560014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.566014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.582015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591185] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.610015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.631015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.642834] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.645356] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.648408] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.650424] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.653236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.658532] iommu: Default domain type: Passthrough [ 0.659318] SCSI subsystem initialized [ 0.660117] ACPI: bus type USB registered [ 0.661099] usbcore: registered new interface driver usbfs [ 0.663079] usbcore: registered new interface driver hub [ 0.665093] usbcore: registered new device driver usb [ 0.666166] pps_core: LinuxPPS API ver. 1 registered [ 0.668022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.671057] PTP clock support registered [ 0.674064] EDAC MC: Ver: 3.0.0 [ 0.677202] PCI: Using ACPI for IRQ routing [ 0.679822] NetLabel: Initializing [ 0.681010] NetLabel: domain hash size = 128 [ 0.682008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.684074] NetLabel: unlabeled traffic allowed by default [ 0.686101] vgaarb: loaded [ 0.688276] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.689011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.694370] clocksource: Switched to clocksource kvm-clock [ 0.797554] VFS: Disk quotas dquot_6.6.0 [ 0.799471] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.802065] *** VALIDATE ramfs *** [ 0.803397] *** VALIDATE hugetlbfs *** [ 0.804920] pnp: PnP ACPI init [ 0.807344] pnp: PnP ACPI: found 6 devices [ 0.829664] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.833131] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.834752] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.836752] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.839024] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.841180] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.843946] NET: Registered protocol family 2 [ 0.846082] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.851146] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.855137] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.861033] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.864784] TCP: Hash tables configured (established 65536 bind 65536) [ 0.867356] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.870364] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873516] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.876902] NET: Registered protocol family 1 [ 0.879988] RPC: Registered named UNIX socket transport module. [ 0.882337] RPC: Registered udp transport module. [ 0.884261] RPC: Registered tcp transport module. [ 0.886092] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.888719] NET: Registered protocol family 44 [ 0.890747] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.893224] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.895392] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.898112] PCI: CLS 0 bytes, default 64 [ 0.899508] Unpacking initramfs... [ 2.278555] debug: unmapping init [mem 0xffff89d97cc54000-0xffff89d97ffbffff] [ 2.282991] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.285124] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.287256] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.799948] Initialise system trusted keyrings [ 2.802741] Key type blacklist registered [ 2.805320] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.816425] zbud: loaded [ 2.819743] *** VALIDATE nfs *** [ 2.821200] *** VALIDATE nfs4 *** [ 2.822874] pstore: using deflate compression [ 2.825971] Platform Keyring initialized [ 2.938464] NET: Registered protocol family 38 [ 2.940754] Key type asymmetric registered [ 2.942663] Asymmetric key parser 'x509' registered [ 2.944514] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.947762] io scheduler mq-deadline registered [ 2.949718] io scheduler kyber registered [ 2.953358] io scheduler bfq registered [ 2.955344] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.958877] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.962269] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.965740] ACPI: Power Button [PWRF] [ 2.972166] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.979935] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.995487] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.004733] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.019547] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.048256] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.076578] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.081667] Non-volatile memory driver v1.3 [ 3.083076] Linux agpgart interface v0.103 [ 3.113710] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.116203] vda: detected capacity change from 0 to 77115392 [ 3.130279] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.133231] vdb: detected capacity change from 0 to 1073741824 [ 3.148789] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.152075] vdc: detected capacity change from 0 to 2621440000 [ 3.168416] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.171674] vdd: detected capacity change from 0 to 2621440000 [ 3.185198] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.188387] vde: detected capacity change from 0 to 4294967296 [ 3.203762] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.207284] vdf: detected capacity change from 0 to 4294967296 [ 3.220902] libphy: Fixed MDIO Bus: probed [ 3.230311] usbcore: registered new interface driver usbserial_generic [ 3.233902] usbserial: USB Serial support registered for generic [ 3.237247] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.243790] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.246085] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.248717] mousedev: PS/2 mouse device common for all mice [ 3.252070] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.255764] rtc_cmos 00:05: RTC can wake from S4 [ 3.259081] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.264845] rtc_cmos 00:05: registered as rtc0 [ 3.267159] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.268731] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.270297] intel_pstate: CPU model not supported [ 3.276984] hid: raw HID events driver (C) Jiri Kosina [ 3.279545] usbcore: registered new interface driver usbhid [ 3.282102] usbhid: USB HID core driver [ 3.283851] drop_monitor: Initializing network drop monitor service [ 3.286559] Initializing XFRM netlink socket [ 3.288835] NET: Registered protocol family 10 [ 3.291707] Segment Routing with IPv6 [ 3.293360] NET: Registered protocol family 17 [ 3.295653] mpls_gso: MPLS GSO support [ 3.301484] RAS: Correctable Errors collector initialized. [ 3.303873] AVX version of gcm_enc/dec engaged. [ 3.305615] AES CTR mode by8 optimization enabled [ 3.382590] sched_clock: Marking stable (3382560380, 0)->(4326250059, -943689679) [ 3.386882] registered taskstats version 1 [ 3.389733] Loading compiled-in X.509 certificates [ 3.392260] zswap: loaded using pool lzo/zbud [ 3.416864] Key type big_key registered [ 3.428360] Key type encrypted registered [ 3.430057] ima: No TPM chip found, activating TPM-bypass! [ 3.432272] ima: Allocated hash algorithm: sha1 [ 3.434053] ima: No architecture policies found [ 3.436032] evm: Initialising EVM extended attributes: [ 3.438208] evm: security.selinux [ 3.440444] evm: security.ima [ 3.441660] evm: security.capability [ 3.443097] evm: HMAC attrs: 0x1 [ 3.445779] rtc_cmos 00:05: setting system clock to 2026-10-02 13:12:43 UTC (1790946763) [ 3.452729] debug: unmapping init [mem 0xffffffff88c03000-0xffffffff88dfffff] [ 3.456306] debug: unmapping init [mem 0xffffffff87982000-0xffffffff87c58fff] [ 3.464268] Write protecting the kernel read-only data: 28672k [ 3.467704] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 3.469944] debug: unmapping init [mem 0xffffffff86914000-0xffffffff869fffff] [ 3.502749] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.511655] systemd[1]: Detected virtualization kvm. [ 3.513975] systemd[1]: Detected architecture x86-64. [ 3.515430] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.547431] systemd[1]: No hostname configured. [ 3.548964] systemd[1]: Set hostname to . [ 3.551059] random: systemd: uninitialized urandom read (16 bytes read) [ 3.553231] systemd[1]: Initializing machine ID from random generator. [ 3.609904] random: ln: uninitialized urandom read (6 bytes read) [ 3.734070] random: systemd: uninitialized urandom read (16 bytes read) [ 3.737238] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.742981] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.748197] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.438336] device-mapper: uevent: version 1.0.3 [ 4.440894] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] [ 5.099967] random: fast init done Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.157246] virtio_net virtio0 ens2: renamed from eth0 [ 5.217729] scsi host0: ata_piix [ 5.249730] scsi host1: ata_piix [ 5.251228] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.253821] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.956420] random: crng init done [ 9.958367] random: 7 urandom warning(s) missed due to ratelimiting [ 10.036539] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.256844] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.597713] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.435601] SELinux: Disabled at runtime. [ 14.541597] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.553315] systemd[1]: Detected virtualization kvm. [ 14.555594] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.876423] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.884696] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.897632] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.904697] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.910794] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.931791] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.949585] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ 16.317053] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 17.165845] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.830260] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.190815] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.325569] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.383883] EDAC sbridge: Ver: 1.1.2 [ 22.397782] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 23.017551] NFS: Registering the id_resolver key type [ 23.020129] Key type id_resolver registered [ 23.026717] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg608-server login: [ 57.173382] hrtimer: interrupt took 4322431 ns [ 76.315393] libcfs: loading out-of-tree module taints kernel. [ 76.356199] Key type ._llcrypt registered [ 76.358686] Key type .llcrypt registered [ 76.467902] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 93.692790] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 95.006707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.019111] alg: No test for adler32 (adler32-zlib) [ 96.464180] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 97.351600] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 99.071201] Key type lgssc registered [ 100.706823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.173216] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.695717] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 166.610522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 166.638381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.807134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.835490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.920804] Lustre: lustre-MDT0000: new disk, initializing [ 167.997650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.012404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.719721] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.481062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.584177] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.611730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.616303] Lustre: Skipped 1 previous similar message [ 184.700612] Lustre: lustre-MDT0001: new disk, initializing [ 184.766663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.801305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.815717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.027791] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.685688] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 202.047987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.336831] Lustre: lustre-OST0000: new disk, initializing [ 202.344757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.351610] Lustre: 8449:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.431586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.454917] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.478396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.512347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.599078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.301689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.469604] Lustre: lustre-OST0001: new disk, initializing [ 222.479847] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 222.489421] Lustre: 9522:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.535440] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.432371] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.418387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.428445] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.558293] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 240.321402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.680602] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 253.525673] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.416146] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing yml_node [ 261.803278] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 264.165245] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 266.609881] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 268.243743] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 09:17:06 EDT 2026 [ 284.543432] Lustre: DEBUG MARKER: only running tests: 33-75 [ 286.101412] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 287.447973] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 302.560319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 302.570321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.589636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.143700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.147461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.154098] Lustre: Skipped 1 previous similar message [ 306.172326] Lustre: Skipped 1 previous similar message [ 308.576646] Lustre: server umount lustre-MDT0000 complete [ 311.704843] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947071 with bad export cookie 12379331566109543842 [ 311.706427] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.714057] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 316.384191] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.385388] LustreError: 6520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.396421] Lustre: Skipped 1 previous similar message [ 316.397147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.408988] LustreError: 6520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 316.444602] Lustre: Skipped 3 previous similar messages [ 321.506177] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.508086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.521336] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 321.544135] Lustre: Skipped 1 previous similar message [ 326.111126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 326.378546] Lustre: server umount lustre-MDT0001 complete [ 330.462426] Lustre: server umount lustre-OST0000 complete [ 334.181187] Lustre: server umount lustre-OST0001 complete [ 340.846851] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 350.770776] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 385.030274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.218667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 385.259499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 385.331640] Lustre: lustre-MDT0000: new disk, initializing [ 385.398307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 385.416468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 389.553240] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 399.021863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.124554] Lustre: 15126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.173845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.178122] Lustre: Skipped 1 previous similar message [ 399.262253] Lustre: lustre-MDT0001: new disk, initializing [ 399.356892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.411487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 399.424083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 403.136451] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 411.398601] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 417.419593] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.806523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 425.135843] Lustre: lustre-OST0000: new disk, initializing [ 425.144388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.163644] Lustre: 17095:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.275841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.606992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.613709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.712860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.732335] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 441.472957] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 446.683558] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 446.900446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.415252] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 451.625476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.703436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.704173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.710578] LustreError: Skipped 1 previous similar message [ 457.712354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.744276] Lustre: Skipped 2 previous similar messages [ 459.873742] Lustre: server umount lustre-OST0000 complete [ 465.888625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 465.895241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.909655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 465.915368] Lustre: Skipped 1 previous similar message [ 469.859414] Lustre: server umount lustre-MDT0000 complete [ 473.056802] LustreError: 15139:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.081897] LustreError: 15139:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 473.937582] LustreError: 15120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947233 with bad export cookie 12379331566109545375 [ 473.938410] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 473.947126] LustreError: 15120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 474.350822] Lustre: server umount lustre-MDT0001 complete [ 478.140248] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:20:36 (1790947236) [ 480.379759] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 481.922781] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:20:40 (1790947240) [ 487.597243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.017272] LustreError: 19504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.098169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.906700] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 493.538554] LustreError: 19505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.789599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.524668] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 512.329037] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 518.323625] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 525.063552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 531.161095] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 538.350557] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 546.784604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 546.800062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.811913] Lustre: Skipped 1 previous similar message [ 546.825116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 546.830446] Lustre: Skipped 2 previous similar messages [ 551.263776] Lustre: server umount lustre-OST0000 complete [ 556.513627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 556.529871] LustreError: Skipped 1 previous similar message [ 561.010354] Lustre: server umount lustre-MDT0000 complete [ 564.197645] LustreError: 19504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.210961] LustreError: 19504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 564.853826] LustreError: 19486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947324 with bad export cookie 12379331566109546257 [ 564.855428] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.862508] LustreError: 19486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 565.331547] Lustre: server umount lustre-MDT0001 complete [ 577.438817] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 580.667217] Key type lgssc unregistered [ 581.021501] LNet: 23036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.026392] LNetError: 23036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.056795] LNet: Removed LNI 192.168.206.108@tcp [ 582.251317] Key type .llcrypt unregistered [ 582.256679] Key type ._llcrypt unregistered [ 604.262658] Key type ._llcrypt registered [ 604.265197] Key type .llcrypt registered [ 604.391427] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 605.569546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 605.597487] alg: No test for adler32 (adler32-zlib) [ 606.630925] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 606.882695] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 608.551205] Key type lgssc registered [ 609.887935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 617.820237] loop: module loaded [ 617.880348] loop0: detected capacity change from 0 to 204800000 [ 618.118526] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 618.134478] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 618.351418] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 623.523750] loop0: detected capacity change from 0 to 204800000 [ 623.681117] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 623.690137] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 623.702734] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.213807] loop0: detected capacity change from 0 to 204800000 [ 632.904176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 632.929071] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.144760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 634.207743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 634.281406] Lustre: lustre-MDT0000: new disk, initializing [ 634.346909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.374867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 638.912654] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 646.928988] loop1: detected capacity change from 0 to 204800000 [ 651.805657] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 652.012961] Lustre: lustre-OST07c6: new disk, initializing [ 652.016056] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 652.020054] Lustre: Skipped 1 previous similar message [ 652.026542] Lustre: 26238:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 652.136636] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 657.913301] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 662.103101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 662.111903] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 662.150786] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 668.278806] Lustre: 26245:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 668.284125] Lustre: 26245:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 668.288383] Lustre: 26245:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 668.293706] Lustre: 26245:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 668.298317] Lustre: 26245:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 668.304818] Lustre: 26245:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 684.562940] Lustre: 25227:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 684.570664] Lustre: 25227:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 684.575949] Lustre: 25227:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 684.582371] Lustre: 25227:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 684.589273] Lustre: 25227:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 684.593952] Lustre: 25227:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 689.698221] Lustre: 25227:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 689.707953] Lustre: 25227:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 689.713238] Lustre: 25227:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 689.718828] Lustre: 25227:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.732241] Lustre: 25227:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 689.749361] Lustre: 25227:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.764282] Lustre: 25227:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 689.773792] Lustre: 25227:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.787076] Lustre: 25227:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 689.796574] Lustre: 25227:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.805183] Lustre: 25227:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 689.811125] Lustre: 25227:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.031757] Lustre: 25226:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 693.038473] Lustre: 25226:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 693.043107] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 693.050380] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.055706] Lustre: 25226:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.067836] Lustre: 25226:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.079149] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 693.089687] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.101185] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 693.109474] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.120229] Lustre: 25226:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 693.126235] Lustre: 25226:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 697.641687] Lustre: 25225:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 697.650531] Lustre: 25225:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 697.658594] Lustre: 25225:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 697.674123] Lustre: 25225:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 697.687922] Lustre: 25225:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 697.703795] Lustre: 25225:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 697.713320] Lustre: 25225:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 697.724697] Lustre: 25225:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 697.746569] Lustre: 25225:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 697.755455] Lustre: 25225:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 697.767029] Lustre: 25225:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 697.780501] Lustre: 25225:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.914028] Lustre: 26245:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 708.918396] Lustre: 26245:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 708.922400] Lustre: 26245:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 708.925779] Lustre: 26245:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 708.929200] Lustre: 26245:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 708.933068] Lustre: 26245:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 708.936469] Lustre: 26245:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 708.940304] Lustre: 26245:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 708.944255] Lustre: 26245:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 708.947682] Lustre: 26245:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 708.951093] Lustre: 26245:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.954711] Lustre: 26245:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.269532] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 759.269663] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.294280] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 761.904490] Lustre: server umount lustre-OST07c6 complete [ 768.776147] Lustre: server umount lustre-MDT0000 complete [ 781.647637] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 784.896515] Key type lgssc unregistered [ 785.341896] LNet: 28794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.360303] LNetError: 28794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.391037] LNet: Removed LNI 192.168.206.108@tcp [ 786.625366] Key type .llcrypt unregistered [ 786.629086] Key type ._llcrypt unregistered [ 806.015452] Key type ._llcrypt registered [ 806.019970] Key type .llcrypt registered [ 806.193914] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 807.776221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 807.827880] alg: No test for adler32 (adler32-zlib) [ 808.920611] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 809.159845] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 810.807555] Key type lgssc registered [ 811.820237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 818.317620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 818.331979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.639792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 822.804633] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 829.233579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 829.521209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 832.593271] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 837.776670] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 841.889236] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 846.270180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.538131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.444549] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 855.272306] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 865.013104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 880.609818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.616132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.618565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.635278] Lustre: Skipped 1 previous similar message [ 883.682774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 883.700562] Lustre: Skipped 2 previous similar messages [ 885.225106] Lustre: server umount lustre-OST0000 complete [ 890.851725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 890.864114] LustreError: Skipped 1 previous similar message [ 890.871577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.901354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 893.921445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 893.922523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.380817] Lustre: server umount lustre-MDT0000 complete [ 899.042421] LustreError: 31768:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 899.051512] LustreError: 31768:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 899.966285] LustreError: 30061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947660 with bad export cookie 15313540117011793550 [ 899.968110] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 899.982117] LustreError: 30061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 900.365567] Lustre: server umount lustre-MDT0001 complete [ 910.942706] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 913.760779] Key type lgssc unregistered [ 914.113478] LNet: 33838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.118662] LNetError: 33838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.137062] LNet: Removed LNI 192.168.206.108@tcp [ 914.819368] Key type .llcrypt unregistered [ 914.821567] Key type ._llcrypt unregistered [ 931.589386] Key type ._llcrypt registered [ 931.591799] Key type .llcrypt registered [ 931.682820] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 932.784123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 932.807269] alg: No test for adler32 (adler32-zlib) [ 933.771314] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 933.963986] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 935.631191] Key type lgssc registered [ 936.511198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 942.762248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 942.809780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.160198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.097149] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 954.349682] LustreError: 35123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.367964] LustreError: 35123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 955.234204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.600972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 959.795961] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 966.898411] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 972.215756] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 977.805964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 978.093627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.490798] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 989.369069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 989.784788] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 999.911738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.918568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.925548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.933270] Lustre: Skipped 1 previous similar message [ 999.947910] Lustre: Skipped 1 previous similar message [ 1004.677560] Lustre: server umount lustre-OST0000 complete [ 1010.144321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.147201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1010.159639] Lustre: Skipped 1 previous similar message [ 1013.886502] Lustre: server umount lustre-MDT0000 complete [ 1015.270087] LustreError: 35123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.295325] LustreError: 35123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1016.578875] LustreError: 35105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947776 with bad export cookie 13214935945178752691 [ 1016.584215] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.586612] LustreError: 35105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1016.792576] Lustre: server umount lustre-MDT0001 complete [ 1026.583933] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1028.718391] Key type lgssc unregistered [ 1028.935631] LNet: 38655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.944360] LNetError: 38655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.969579] LNet: Removed LNI 192.168.206.108@tcp [ 1029.676592] Key type .llcrypt unregistered [ 1029.680640] Key type ._llcrypt unregistered [ 1050.596318] Key type ._llcrypt registered [ 1050.598818] Key type .llcrypt registered [ 1050.723867] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1051.399985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1051.504136] alg: No test for adler32 (adler32-zlib) [ 1052.539883] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 1052.756418] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1054.439331] Key type lgssc registered [ 1055.783280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1064.280299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1064.296539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.666215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.430561] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1075.170133] LustreError: 40088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.182810] LustreError: 40088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1078.670393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.910621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1083.397794] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1090.309611] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1095.049761] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1100.854554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1101.245240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.892211] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1110.469505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1114.245362] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1119.715519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1119.725524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.737328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.935641] Lustre: server umount lustre-MDT0000 complete [ 1120.741624] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.742165] LustreError: 40816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.751225] Lustre: Skipped 1 previous similar message [ 1120.778958] LustreError: 40816:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1123.568571] LustreError: 40069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947883 with bad export cookie 13359390251011458931 [ 1123.591615] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.857115] LustreError: 40083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.861538] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.882572] LustreError: 40083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1125.895979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1128.864280] LustreError: 40083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.864938] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1128.886051] LustreError: 40083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1128.908453] Lustre: Skipped 1 previous similar message [ 1130.043625] Lustre: server umount lustre-MDT0001 complete [ 1151.534244] Lustre: server umount lustre-OST0000 complete [ 1165.013798] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1167.483468] Key type lgssc unregistered [ 1167.773839] LNet: 43945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.779495] LNetError: 43945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.800088] LNet: Removed LNI 192.168.206.108@tcp [ 1168.655869] Key type .llcrypt unregistered [ 1168.665782] Key type ._llcrypt unregistered [ 1189.263188] Key type ._llcrypt registered [ 1189.269875] Key type .llcrypt registered [ 1189.392260] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1190.236396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1190.335718] alg: No test for adler32 (adler32-zlib) [ 1191.418495] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 1191.630705] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1193.303230] Key type lgssc registered [ 1194.398432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1200.682209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1200.696463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.092231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1205.785712] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1212.388597] LustreError: 45355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.402485] LustreError: 45355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1212.856256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.208278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1216.686352] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1224.328128] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1229.922667] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1236.784581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1237.150176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.863721] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1247.407993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1248.614193] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1256.349506] Lustre: lustre-OST0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 1257.954614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.956895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1257.967946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1257.983190] Lustre: Skipped 1 previous similar message [ 1261.142746] Lustre: server umount lustre-OST0000 complete [ 1279.969160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1279.976628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.997050] Lustre: Skipped 1 previous similar message [ 1280.004890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1283.562900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.562939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1283.600217] Lustre: Skipped 1 previous similar message [ 1285.332762] Lustre: server umount lustre-MDT0000 complete [ 1288.673924] LustreError: 45355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.700201] LustreError: 45355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1288.764594] LustreError: 45337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948048 with bad export cookie 9455796284815973958 [ 1288.777214] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1288.787332] LustreError: 45337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1295.294943] Lustre: server umount lustre-MDT0001 complete [ 1304.389236] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1306.994176] Key type lgssc unregistered [ 1307.259300] LNet: 49062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1307.270856] LNetError: 49062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1307.302728] LNet: Removed LNI 192.168.206.108@tcp [ 1307.947243] Key type .llcrypt unregistered [ 1307.951740] Key type ._llcrypt unregistered [ 1326.982531] Key type ._llcrypt registered [ 1326.985101] Key type .llcrypt registered [ 1327.091828] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1327.807278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1327.922595] alg: No test for adler32 (adler32-zlib) [ 1328.995877] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 1329.241859] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1330.895215] Key type lgssc registered [ 1332.000494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1338.659996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1338.676392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.140505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.673751] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1350.636802] LustreError: 50473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.667910] LustreError: 50473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1351.755949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.049731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1355.077412] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1361.923489] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1367.673659] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1374.659741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1374.949578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.819555] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1385.204556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1387.970754] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1394.344642] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1397.497992] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1409.573146] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-10-02 9h36m07s [ 1410.966285] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1413.601141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1413.614701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.630533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.694853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.702882] Lustre: Skipped 2 previous similar messages [ 1416.613080] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 1416.619241] Lustre: Skipped 2 previous similar messages [ 1419.214979] Lustre: server umount lustre-MDT0000 complete [ 1420.780464] LustreError: 52964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.800953] LustreError: 52964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1422.247647] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1422.724567] LustreError: 50468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.925494] LustreError: 51761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.942865] LustreError: 51761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1429.081551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.238511] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1429.467289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.541492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1430.519872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1433.715426] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1437.627938] Lustre: DEBUG MARKER: Wait for df (36905) ... [ 1439.258428] Lustre: DEBUG MARKER: done [ 1443.808676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.817964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.825193] Lustre: Skipped 2 previous similar messages [ 1443.829066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1443.831644] Lustre: Skipped 1 previous similar message [ 1444.832212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.843289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.292751] Lustre: server umount lustre-OST0000 complete [ 1456.097367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1456.099767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.108882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.122836] Lustre: Skipped 1 previous similar message [ 1456.128605] Lustre: Skipped 3 previous similar messages [ 1467.359269] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1469.411991] LustreError: 50469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.429823] LustreError: 50469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1469.513388] Lustre: server umount lustre-MDT0000 complete [ 1473.095634] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948233 with bad export cookie 6780051029678836336 [ 1473.116946] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1473.122389] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1473.402436] Lustre: server umount lustre-MDT0001 complete [ 1483.003489] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1485.742933] Key type lgssc unregistered [ 1485.992808] LNet: 55394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.998666] LNetError: 55394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.019906] LNet: Removed LNI 192.168.206.108@tcp [ 1486.950360] Key type .llcrypt unregistered [ 1486.954153] Key type ._llcrypt unregistered [ 1521.604510] Key type ._llcrypt registered [ 1521.607815] Key type .llcrypt registered [ 1521.710650] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1522.627778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1522.790716] alg: No test for adler32 (adler32-zlib) [ 1523.947537] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 1524.295092] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1526.047178] Key type lgssc registered [ 1527.069673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1534.325430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1534.343831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.628748] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1535.645469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1535.829476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.832530] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1548.030108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.166396] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1548.203196] Lustre: 57597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1548.483386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1553.222526] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1560.166699] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1565.829141] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1571.955917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.090987] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1572.281662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.979676] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1579.075554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1586.123518] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1591.798098] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-10-02 9h39m10s [ 1593.211177] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1600.174252] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1602.751312] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1602.935959] LustreError: 57604:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1623.451492] Lustre: lustre-MDT0000: Client 4f6db7d0-acc1-4ff4-8a85-a1c42984557b (at 192.168.206.8@tcp) reconnecting [ 1643.031118] LustreError: 57604:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1645.263316] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1647.321940] Lustre: DEBUG MARKER: done [ 1655.465415] Lustre: server umount lustre-OST0000 complete [ 1660.897762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.918588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1660.926162] Lustre: Skipped 1 previous similar message [ 1661.999574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.013531] Lustre: Skipped 1 previous similar message [ 1668.133430] Lustre: server umount lustre-MDT0000 complete [ 1671.142089] Lustre: 56913:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948415/real 1790948415] req@0000000065b78c90 x1877945387898368/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790948431 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1671.166884] LustreError: 57590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948431 with bad export cookie 5262611589871222197 [ 1671.170412] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1671.172558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.172566] Lustre: Skipped 1 previous similar message [ 1671.222220] LustreError: 57590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1671.469482] Lustre: server umount lustre-MDT0001 complete [ 1681.607184] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1684.378643] Key type lgssc unregistered [ 1684.658065] LNet: 61932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.681728] LNetError: 61932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.733864] LNet: Removed LNI 192.168.206.108@tcp [ 1686.380878] Key type .llcrypt unregistered [ 1686.383079] Key type ._llcrypt unregistered [ 1727.021290] Key type ._llcrypt registered [ 1727.024025] Key type .llcrypt registered [ 1727.186979] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1728.577373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.597516] alg: No test for adler32 (adler32-zlib) [ 1729.604382] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 1729.837158] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1731.543597] Key type lgssc registered [ 1732.916661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.079819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1736.118453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.426518] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1737.466793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1737.702648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.589936] Lustre: Failing over lustre-MDT0000 [ 1738.862140] Lustre: server umount lustre-MDT0000 complete [ 1745.302392] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:41:43 (1790948503) [ 1752.085526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.593294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.453203] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1765.694094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.785688] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1765.810388] Lustre: 64747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1766.148832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1770.685790] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1777.821995] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1784.347069] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1791.636206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1791.797367] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1792.057217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1798.327186] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1801.930601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1807.025399] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1813.073625] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1822.691686] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.695459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1822.709487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1822.721454] Lustre: Skipped 1 previous similar message [ 1826.215247] Lustre: server umount lustre-MDT0000 complete [ 1830.027069] LustreError: 64739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948590 with bad export cookie 14209966645560604232 [ 1830.035801] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.042964] LustreError: 64739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1830.408303] Lustre: server umount lustre-MDT0001 complete [ 1834.175962] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1839.511079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1848.160081] Lustre: 63662:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948592/real 1790948592] req@000000007897c02e x1877945602842368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790948608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1848.200269] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.220983] Lustre: Skipped 2 previous similar messages [ 1848.244902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.359312] Lustre: 63661:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948595/real 1790948595] req@0000000022612752 x1877945602842496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790948611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1854.431710] Lustre: 63661:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948598/real 1790948598] req@00000000b9c18198 x1877945602842624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790948614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1855.393907] LustreError: 63659:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000da0919ee x1877945602843392/t0(0) o250->MGC192.168.206.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1855.704222] LustreError: 66596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.721691] LustreError: 66596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1855.807235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.940854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1856.803050] Lustre: 63662:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948600/real 1790948600] req@000000005056e71e x1877945602842880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790948616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1857.840521] LustreError: 66597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.849859] LustreError: 68520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1858.867136] LustreError: 68520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1860.546786] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1861.920906] Lustre: 63663:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790948605/real 1790948605] req@0000000009c033d6 x1877945602843008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790948621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1864.166404] LustreError: 68519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.189149] LustreError: 68519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1869.282422] LustreError: 68521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.294665] LustreError: 68521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1869.325110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.670258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1873.639493] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1874.929853] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1877.038434] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1881.014720] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1886.269581] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1892.947467] Lustre: server umount lustre-MDT0000 complete [ 1895.392319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1895.410840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.431796] LustreError: 69766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.829593] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948656 with bad export cookie 14209966645560606507 [ 1896.850805] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1896.854037] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1897.457160] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.477784] Lustre: Skipped 1 previous similar message [ 1897.486482] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1899.350048] Lustre: server umount lustre-MDT0001 complete [ 1906.017107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1916.357410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.209991] LustreError: 66596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.229353] LustreError: 66596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1922.277067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.318555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1927.627920] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1936.878826] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1938.680233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.156475] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1944.558038] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1952.240780] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1959.085212] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1970.147737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1970.151109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.154434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.182973] Lustre: Skipped 2 previous similar messages [ 1972.000589] Lustre: server umount lustre-MDT0000 complete [ 1975.268531] LustreError: 66597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.290894] LustreError: 66597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 1975.347561] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948735 with bad export cookie 14209966645560607886 [ 1975.350341] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.360363] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1975.670708] Lustre: server umount lustre-MDT0001 complete [ 1981.817501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1985.072477] Lustre: DEBUG MARKER: files compared the same [ 1996.354566] Lustre: server umount lustre-OST0000 complete [ 2013.153777] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2016.280889] Key type lgssc unregistered [ 2016.605906] LNet: 74475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.611254] LNetError: 74475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.629896] LNet: Removed LNI 192.168.206.108@tcp [ 2017.637253] Key type .llcrypt unregistered [ 2017.638785] Key type ._llcrypt unregistered [ 2041.058832] Key type ._llcrypt registered [ 2041.062253] Key type .llcrypt registered [ 2041.152419] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2041.972811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2041.997794] alg: No test for adler32 (adler32-zlib) [ 2043.005869] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 2043.191052] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2044.831255] Key type lgssc registered [ 2046.082907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2053.687645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2053.701160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.103635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.921893] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug +malloc all [ 2065.378686] LustreError: 75911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.400334] LustreError: 75911:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2068.862711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.136168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.167201] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug +malloc all [ 2081.519660] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2087.807512] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2094.124854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.440673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.849584] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug +malloc all [ 2100.595816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2106.767766] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2125.287398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.289144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2125.299070] Lustre: Skipped 1 previous similar message [ 2125.316419] Lustre: Skipped 1 previous similar message [ 2129.572740] Lustre: server umount lustre-OST0000 complete [ 2135.525159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2135.527239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.534841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.549279] Lustre: Skipped 1 previous similar message [ 2135.569885] Lustre: Skipped 1 previous similar message [ 2139.662906] Lustre: server umount lustre-MDT0000 complete [ 2140.640587] LustreError: 75907:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.666123] LustreError: 75907:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2143.747888] LustreError: 75893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948903 with bad export cookie 12105983663750980997 [ 2143.752577] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2143.758579] LustreError: 75893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2144.124329] Lustre: server umount lustre-MDT0001 complete [ 2155.781180] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2158.702350] Key type lgssc unregistered [ 2158.988989] LNet: 79464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.995659] LNetError: 79464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2159.016242] LNet: Removed LNI 192.168.206.108@tcp [ 2159.725774] Key type .llcrypt unregistered [ 2159.728413] Key type ._llcrypt unregistered [ 2183.542555] Key type ._llcrypt registered [ 2183.549443] Key type .llcrypt registered [ 2183.653120] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2184.777534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2184.803683] alg: No test for adler32 (adler32-zlib) [ 2185.934304] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 2186.159543] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2187.839200] Key type lgssc registered [ 2189.280826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2196.823400] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2196.845904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2212.383457] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2217.503294] LustreError: 80869:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.108@tcp: failed processing log, type 4: rc = -110 [ 2244.128470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.706471] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2258.921422] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2266.404567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.548968] Lustre: *** cfs_fail_loc=706, val=0*** [ 2267.018819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.097581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2271.462000] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2272.235988] LustreError: 80895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.266066] LustreError: 80895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2277.351704] LustreError: 80894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.385271] LustreError: 81954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.412859] LustreError: 81954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2279.807106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.184866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2284.389279] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2291.753415] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2298.403173] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2300.901698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2300.906146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.910541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2300.947302] Lustre: Skipped 1 previous similar message [ 2306.019892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.037326] Lustre: Skipped 2 previous similar messages [ 2306.804797] Lustre: server umount lustre-OST0000 complete [ 2311.138418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.142203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2311.142296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.157316] Lustre: Skipped 1 previous similar message [ 2316.257083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.263521] Lustre: Skipped 2 previous similar messages [ 2316.683726] Lustre: server umount lustre-MDT0000 complete [ 2319.888305] LustreError: 83731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949079 with bad export cookie 12840823446119920828 [ 2319.890985] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.900669] LustreError: 83731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2320.217820] Lustre: server umount lustre-MDT0001 complete [ 2331.085480] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2333.953826] Key type lgssc unregistered [ 2334.323668] LNet: 84483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.328965] LNetError: 84483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.343252] LNet: Removed LNI 192.168.206.108@tcp [ 2335.136190] Key type .llcrypt unregistered [ 2335.141226] Key type ._llcrypt unregistered [ 2358.643488] Key type ._llcrypt registered [ 2358.645961] Key type .llcrypt registered [ 2358.762941] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2359.558373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2359.582793] alg: No test for adler32 (adler32-zlib) [ 2360.632927] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 2360.947571] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2362.632923] Key type lgssc registered [ 2363.758721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2371.158611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2371.210257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.843852] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2383.335381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.706748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2387.774943] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2388.963745] LustreError: 86602:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.981556] LustreError: 86602:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2393.056923] LustreError: 86605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.068239] LustreError: 86605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2395.906589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.217430] LustreError: 86602:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.241552] LustreError: 86602:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2396.250657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.260500] LustreError: 86606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.274165] LustreError: 86606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2401.728752] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2406.371364] LustreError: 86605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.382138] LustreError: 86605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2409.908417] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.919072] Lustre: MGS: Client 858d34a3-614f-4275-9ff1-aa197986039d (at 0@lo) reconnecting [ 2409.944439] LustreError: 86616:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2410.176796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.251379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2415.299466] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2430.954726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.958697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.964444] Lustre: Skipped 1 previous similar message [ 2430.966437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.596821] Lustre: server umount lustre-OST0000 complete [ 2441.186095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.198048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2441.204851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.230708] Lustre: Skipped 2 previous similar messages [ 2444.687197] Lustre: server umount lustre-MDT0000 complete [ 2446.305417] LustreError: 88146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.336563] LustreError: 88146:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2449.482776] Lustre: server umount lustre-MDT0001 complete [ 2453.532144] Lustre: server umount MGS complete [ 2464.494730] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:53:43 (1790949223) [ 2495.282851] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2505.085431] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2545.675206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.078352] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2559.635735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.878572] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2559.932391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2560.022558] Lustre: lustre-MDT0001: new disk, initializing [ 2560.111532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2560.130559] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2560.137608] Lustre: 93775:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2560.144518] LustreError: 93775:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2564.745885] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2574.753887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.032367] Lustre: lustre-OST0000: new disk, initializing [ 2575.035940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2575.043990] Lustre: 94541:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2575.116855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.153247] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2585.184222] Lustre: 95014:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b3f743f9 x1877946264493696/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790949361 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2590.763888] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2598.054780] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.063787] Lustre: MGS: Client f871fbf9-d056-4d3d-867d-92b11facdb6f (at 0@lo) reconnecting [ 2598.115687] Lustre: 95587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2598.142220] LustreError: 93760:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2598.149743] LustreError: 93760:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2598.190770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2598.293944] Lustre: lustre-MDT0000: new disk, initializing [ 2598.506218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.565561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2598.679961] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2599.652259] Lustre: 95611:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001666c6c7 x1877946264500352/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790949375 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2602.915491] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2605.536296] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2608.748615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2608.762537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2608.900633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2633.189213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.203784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2633.208633] Lustre: Skipped 1 previous similar message [ 2633.210474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2638.287444] Lustre: server umount lustre-OST0000 complete [ 2643.425362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2643.437485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.451892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.473714] Lustre: Skipped 2 previous similar messages [ 2643.497725] Lustre: Skipped 2 previous similar messages [ 2648.546983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2648.550764] Lustre: Skipped 1 previous similar message [ 2648.689790] Lustre: server umount lustre-MDT0000 complete [ 2659.029527] Lustre: server umount lustre-MDT0001 complete [ 2662.940776] Lustre: server umount MGS complete [ 2675.856409] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:57:13 (1790949433) [ 2694.815476] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2697.759659] Key type lgssc unregistered [ 2698.155341] LNet: 98107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2698.166466] LNetError: 98107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2698.193609] LNet: Removed LNI 192.168.206.108@tcp [ 2699.255536] Key type .llcrypt unregistered [ 2699.260409] Key type ._llcrypt unregistered [ 2717.463418] Key type ._llcrypt registered [ 2717.466277] Key type .llcrypt registered [ 2717.581675] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2718.649595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2718.751435] alg: No test for adler32 (adler32-zlib) [ 2719.898283] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 2720.209773] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2721.959222] Key type lgssc registered [ 2723.336247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2732.024632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2732.038889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.596399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.300419] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2743.786094] LustreError: 99347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.806425] LustreError: 99347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2746.607085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2747.037582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2751.524895] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2759.229536] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2765.182434] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2772.626420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.081582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.223546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2779.726615] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2787.456669] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2799.075459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2799.087947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.103636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.171179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2803.174016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.181769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.211669] Lustre: Skipped 1 previous similar message [ 2808.311368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2808.321053] Lustre: Skipped 1 previous similar message [ 2812.383371] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2812.590510] Lustre: server umount lustre-OST0000 complete [ 2818.532526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.538502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.562494] Lustre: Skipped 1 previous similar message [ 2818.563432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2818.569039] Lustre: Skipped 1 previous similar message [ 2822.614346] Lustre: server umount lustre-MDT0000 complete [ 2823.649042] LustreError: 99346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.685681] LustreError: 99346:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2826.467326] LustreError: 99326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949586 with bad export cookie 4586823440774539585 [ 2826.486146] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2826.493530] LustreError: 99326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2827.081284] Lustre: server umount lustre-MDT0001 complete [ 2838.377879] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2841.326890] Key type lgssc unregistered [ 2841.623637] LNet: 102881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2841.629141] LNetError: 102881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2841.643501] LNet: Removed LNI 192.168.206.108@tcp [ 2842.669324] Key type .llcrypt unregistered [ 2842.671561] Key type ._llcrypt unregistered [ 2860.291631] Key type ._llcrypt registered [ 2860.295217] Key type .llcrypt registered [ 2860.397186] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2861.601502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2861.669271] alg: No test for adler32 (adler32-zlib) [ 2862.753572] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 2863.053655] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2864.743226] Key type lgssc registered [ 2865.933754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2871.735374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2871.754571] LustreError: 103969:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2871.817093] LustreError: 103981:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2871.824502] LustreError: 103969:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4943 [ 2871.832757] LustreError: 103969:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2871.842943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2871.846373] LustreError: 103969:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2873.349336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2881.076543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.468321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2885.896924] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2892.864311] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing lsmod [ 2896.459509] LustreError: 105176:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2896.516356] LustreError: 105189:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2896.520318] LustreError: 105176:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4941 [ 2896.533103] LustreError: 105176:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2896.545419] LustreError: 105176:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2896.549061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.988725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.043801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2901.989892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2902.009563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.018313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.523656] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.525282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.546221] Lustre: Skipped 1 previous similar message [ 2903.561981] Lustre: Skipped 2 previous similar messages [ 2906.501310] Lustre: server umount lustre-MDT0000 complete [ 2906.662804] LustreError: 104014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.693019] LustreError: 104014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2910.028403] LustreError: 103994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949670 with bad export cookie 5460626174982816890 [ 2910.039402] LustreError: 103994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2910.043208] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.431389] Lustre: server umount lustre-MDT0001 complete [ 2924.526978] Lustre: server umount lustre-OST0000 complete [ 2934.060047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.477142] LustreError: 106264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.490162] LustreError: 106264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2934.537251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.007564] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2939.875769] LustreError: 106265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.977192] LustreError: 106264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.743818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.073212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2952.382093] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2959.734795] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2965.658307] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2972.818526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.228113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.370130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 2979.619288] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2986.596715] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2994.670693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.688057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2994.690634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2994.716413] Lustre: Skipped 1 previous similar message [ 2994.837828] Lustre: server umount lustre-OST0000 complete [ 2999.264361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2999.281421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.302533] Lustre: Skipped 1 previous similar message [ 2999.310409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3004.902313] LustreError: 106264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.927614] LustreError: 106264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3004.963565] Lustre: server umount lustre-MDT0000 complete [ 3008.381923] LustreError: 106247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949768 with bad export cookie 5460626174982817828 [ 3008.391409] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3008.402079] LustreError: 106247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3008.741182] Lustre: server umount lustre-MDT0001 complete [ 3021.690714] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 3024.234884] Key type lgssc unregistered [ 3024.545152] LNet: 109795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3024.552750] LNetError: 109795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3024.565331] LNet: Removed LNI 192.168.206.108@tcp [ 3025.397752] Key type .llcrypt unregistered [ 3025.400664] Key type ._llcrypt unregistered [ 3051.884833] Key type ._llcrypt registered [ 3051.886915] Key type .llcrypt registered [ 3052.021156] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3053.648964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3053.671601] alg: No test for adler32 (adler32-zlib) [ 3054.859706] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 3055.362728] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 3057.127181] Key type lgssc registered [ 3058.521291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3066.743286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3066.770539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.177433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.645951] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3078.631820] LustreError: 111233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.648245] LustreError: 111233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3081.148329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.762919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3086.528866] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3094.582149] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3100.526001] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3106.649805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.039138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.127969] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3113.219362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3121.346674] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3126.573606] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3130.596890] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3139.048922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.049052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3139.056162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3139.058613] Lustre: Skipped 1 previous similar message [ 3139.081742] LustreError: Skipped 1 previous similar message [ 3141.711877] Lustre: server umount lustre-OST0000 complete [ 3148.256394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3148.265338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.279852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3148.291763] Lustre: Skipped 1 previous similar message [ 3149.283471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.285323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.382495] Lustre: server umount lustre-MDT0000 complete [ 3154.423822] LustreError: 111230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.456786] LustreError: 111230:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3155.129772] LustreError: 111213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949915 with bad export cookie 1493407014161468513 [ 3155.137451] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.170971] LustreError: 111213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3155.483170] Lustre: server umount lustre-MDT0001 complete [ 3165.545509] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 3168.257752] Key type lgssc unregistered [ 3168.541638] LNet: 114868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3168.552519] LNetError: 114868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3168.572723] LNet: Removed LNI 192.168.206.108@tcp [ 3169.671938] Key type .llcrypt unregistered [ 3169.680391] Key type ._llcrypt unregistered [ 3189.412991] Key type ._llcrypt registered [ 3189.415267] Key type .llcrypt registered [ 3189.538296] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3190.224582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3190.258471] alg: No test for adler32 (adler32-zlib) [ 3191.287169] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 3191.570167] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 3193.239208] Key type lgssc registered [ 3194.241804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3202.472659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3202.489239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3203.930203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.957109] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3213.284079] LustreError: 116105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.316191] LustreError: 116105:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3216.361206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.619883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3221.140036] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3229.355805] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3235.512912] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3241.877769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3242.241275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.582507] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3248.426801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3253.922425] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3272.177280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.179114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3272.179647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3272.200565] Lustre: Skipped 1 previous similar message [ 3277.285789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3277.299555] Lustre: Skipped 1 previous similar message [ 3278.095197] Lustre: server umount lustre-OST0000 complete [ 3282.400554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.410849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3282.411819] Lustre: Skipped 1 previous similar message [ 3282.434709] Lustre: Skipped 2 previous similar messages [ 3287.522202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.532942] Lustre: Skipped 1 previous similar message [ 3288.485238] Lustre: server umount lustre-MDT0000 complete [ 3292.644759] LustreError: 117944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.659233] LustreError: 117944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3292.825445] LustreError: 116087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950052 with bad export cookie 12284197876655587883 [ 3292.828351] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.843431] LustreError: 116087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3293.393617] Lustre: server umount lustre-MDT0001 complete [ 3304.345690] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 3306.913901] Key type lgssc unregistered [ 3307.213519] LNet: 119656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3307.226200] LNetError: 119656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3307.238765] LNet: Removed LNI 192.168.206.108@tcp [ 3308.175315] Key type .llcrypt unregistered [ 3308.179129] Key type ._llcrypt unregistered [ 3330.000694] Key type ._llcrypt registered [ 3330.003498] Key type .llcrypt registered [ 3330.087938] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3331.283119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3331.314047] alg: No test for adler32 (adler32-zlib) [ 3332.371158] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 3332.581571] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 3334.247256] Key type lgssc registered [ 3335.709725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3343.585735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3343.612560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.163455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.546513] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3355.617090] LustreError: 121068:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.646768] LustreError: 121068:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3359.783034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3360.253660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.033155] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3374.561626] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3380.574622] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3388.632604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.219803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.279404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3396.793223] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3405.090972] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3412.816424] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3417.078797] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3417.082771] Lustre: Skipped 2 previous similar messages [ 3419.779368] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3419.826808] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3421.869049] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3421.881486] Lustre: Skipped 2 previous similar messages [ 3428.917933] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3428.921242] Lustre: Skipped 1 previous similar message [ 3433.829944] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.8@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3433.835024] Lustre: Skipped 2 previous similar messages [ 3434.991041] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.8@tcp 192.168.0.[2,10]@tcp [ 3446.919918] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.8/24@tcp in log lustre-MDT0000 [ 3446.925295] Lustre: Skipped 2 previous similar messages [ 3455.641809] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.8/24@tcp [ 3455.656907] Lustre: Skipped 1 previous similar message [ 3465.371714] Lustre: server umount lustre-OST0000 complete [ 3472.368370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3472.387249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3472.395595] Lustre: Skipped 1 previous similar message [ 3477.476300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3477.483769] Lustre: Skipped 1 previous similar message [ 3480.660860] Lustre: server umount lustre-MDT0000 complete [ 3482.594358] LustreError: 121063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.630503] LustreError: 121063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3483.680446] Lustre: 120369:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790950227/real 1790950227] req@00000000121cc076 x1877947283738112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790950243 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3483.714864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3483.749548] Lustre: Skipped 1 previous similar message [ 3483.759920] LustreError: 121064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.783208] LustreError: 121064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3484.851426] LustreError: 121049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950244 with bad export cookie 8330865446977209266 [ 3484.859195] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3484.874255] LustreError: 121049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3485.177430] Lustre: server umount lustre-MDT0001 complete [ 3498.124580] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 3501.535778] Key type lgssc unregistered [ 3501.996901] LNet: 126015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3502.009100] LNetError: 126015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3502.040037] LNet: Removed LNI 192.168.206.108@tcp [ 3502.666167] Key type .llcrypt unregistered [ 3502.668971] Key type ._llcrypt unregistered [ 3532.823202] Key type ._llcrypt registered [ 3532.826561] Key type .llcrypt registered [ 3532.961816] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3534.566768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3534.603017] alg: No test for adler32 (adler32-zlib) [ 3535.700956] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 3535.894319] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 3537.552128] Key type lgssc registered [ 3538.839282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3546.172796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3546.204352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.632472] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3547.649000] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.657946] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.8/24@tcp [ 3551.597201] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3557.873566] LustreError: 127591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.904329] LustreError: 127591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3560.208163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.549367] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3560.575126] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.8/24@tcp [ 3560.619365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3564.478595] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3572.014621] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3577.801601] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3584.656068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3585.196557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.014227] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3592.375690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3600.165264] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3612.135683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3612.143736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.160703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3613.152475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3613.155660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3613.177658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.203508] Lustre: Skipped 1 previous similar message [ 3616.317105] Lustre: server umount lustre-OST0000 complete [ 3622.369356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3622.377510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3622.398148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3626.374253] Lustre: server umount lustre-MDT0000 complete [ 3628.515261] LustreError: 127590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3628.536758] LustreError: 127590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3630.043126] LustreError: 130226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950390 with bad export cookie 1588177554506064675 [ 3630.054767] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3630.061421] LustreError: 130226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3630.550138] Lustre: server umount lustre-MDT0001 complete [ 3642.326893] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 3644.963727] Key type lgssc unregistered [ 3645.215725] LNet: 131177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3645.230364] LNetError: 131177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3645.253797] LNet: Removed LNI 192.168.206.108@tcp [ 3646.184251] Key type .llcrypt unregistered [ 3646.186519] Key type ._llcrypt unregistered [ 3674.709091] Key type ._llcrypt registered [ 3674.711052] Key type .llcrypt registered [ 3674.844639] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 3688.629491] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3689.526170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3689.594518] alg: No test for adler32 (adler32-zlib) [ 3690.608457] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 3690.822518] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 3692.505127] Key type lgssc registered [ 3693.663326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3731.496104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3731.528097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.743485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3732.762260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3732.825704] Lustre: lustre-MDT0000: new disk, initializing [ 3732.941495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3732.962826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3737.998550] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3750.482572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.651798] Lustre: 134597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3750.761098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3750.768892] Lustre: Skipped 1 previous similar message [ 3750.908143] Lustre: lustre-MDT0001: new disk, initializing [ 3751.045508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3751.089104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3751.101761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3755.808843] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3765.481666] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3772.275336] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3780.391537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3780.679703] Lustre: lustre-OST0000: new disk, initializing [ 3780.685531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3780.695398] Lustre: 136566:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3780.841931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.269245] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3790.926502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3790.934708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3791.023386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3799.547171] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3804.826580] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3805.117901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3810.238916] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3810.537689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3816.929481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3816.930120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3816.944620] LustreError: Skipped 1 previous similar message [ 3816.959125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3816.965156] Lustre: Skipped 1 previous similar message [ 3822.052873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.064608] Lustre: Skipped 1 previous similar message [ 3827.168152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3827.169073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.188670] Lustre: Skipped 1 previous similar message [ 3827.433587] Lustre: server umount lustre-OST0000 complete [ 3832.290183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.314972] Lustre: Skipped 1 previous similar message [ 3832.329093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3832.336034] Lustre: Skipped 1 previous similar message [ 3837.534847] Lustre: server umount lustre-MDT0000 complete [ 3840.691948] LustreError: 136565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950600 with bad export cookie 3416713246600662205 [ 3840.694278] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3840.711168] LustreError: 136565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3841.035613] Lustre: server umount lustre-MDT0001 complete [ 3849.134969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.596339] LustreError: 138783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.619458] LustreError: 138783:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3849.677208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3854.028547] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3854.818293] LustreError: 138784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3859.944922] LustreError: 138783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3862.440543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3862.868337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3867.492413] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3876.308762] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3882.384338] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3889.616122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3889.991576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.868791] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3903.949950] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3908.564856] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3908.789740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3912.678485] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3912.845544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3921.330507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3921.642572] Lustre: lustre-OST0001: new disk, initializing [ 3921.653113] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3921.664647] Lustre: 142155:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3921.845530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3927.164882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3927.180671] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3927.264677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3929.307702] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3937.740411] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3938.012083] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3942.738765] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3943.031387] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3949.281375] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff923f58d0b800.ost_server_uuid 50 [ 3951.011436] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff923f58d0b800.ost_server_uuid in IDLE state after 0 sec [ 3961.320125] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.325925] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3961.337025] Lustre: Skipped 2 previous similar messages [ 3961.346681] Lustre: Skipped 1 previous similar message [ 3964.903606] Lustre: server umount lustre-OST0001 complete [ 3966.442179] LustreError: 140604:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.461917] LustreError: 140604:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3968.992935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3968.999418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.043818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3971.057315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3971.068090] Lustre: Skipped 1 previous similar message [ 3971.554635] LustreError: 140604:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.576606] LustreError: 140604:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3974.404475] Lustre: server umount lustre-OST0000 complete [ 3980.768849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3980.774108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.781017] Lustre: Skipped 1 previous similar message [ 3984.389485] Lustre: server umount lustre-MDT0000 complete [ 3986.913961] LustreError: 140846:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.949146] LustreError: 140846:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3987.980046] LustreError: 138764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950748 with bad export cookie 3416713246600663087 [ 3987.981540] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.986541] LustreError: 138764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3988.250594] Lustre: server umount lustre-MDT0001 complete [ 4009.815607] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4012.847112] Key type lgssc unregistered [ 4013.252157] LNet: 145158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4013.262207] LNetError: 145158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4014.312914] LNet: Removed LNI 192.168.206.108@tcp [ 4015.528359] Key type .llcrypt unregistered [ 4015.530629] Key type ._llcrypt unregistered [ 4057.692114] Key type ._llcrypt registered [ 4057.693994] Key type .llcrypt registered [ 4057.779251] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 4074.078588] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4074.917295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4074.982091] alg: No test for adler32 (adler32-zlib) [ 4076.247548] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 4076.515200] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4078.175339] Key type lgssc registered [ 4079.692821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4115.232668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4115.274980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.516848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4116.534834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4116.595302] Lustre: lustre-MDT0000: new disk, initializing [ 4116.643900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.654346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4120.556994] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4129.989428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.123750] Lustre: 149331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4130.178625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4130.181393] Lustre: Skipped 1 previous similar message [ 4130.233764] Lustre: lustre-MDT0001: new disk, initializing [ 4130.316668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4130.340195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4130.360486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4134.431524] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4142.816530] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4148.250986] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4155.908775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.122281] Lustre: lustre-OST0000: new disk, initializing [ 4156.126095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4156.133540] Lustre: 151299:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4156.256657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.631767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4161.653855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4161.810527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4163.592983] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4172.695931] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4177.865859] Lustre: Failing over lustre-OST0000 [ 4178.062663] Lustre: server umount lustre-OST0000 complete [ 4195.300657] Lustre: 147547:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790950939/real 1790950939] req@0000000005ace471 x1877948063843456/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1790950955 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4195.329244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.413334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4196.760119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.793403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.809609] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4198.802829] Lustre: 147547:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790950942/real 1790950942] req@000000000aa72672 x1877948063843840/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790950958 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4198.822612] Lustre: 147547:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4200.816155] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4202.775429] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4202.975445] Lustre: 147545:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790950947/real 1790950947] req@00000000bbf3e14d x1877948063845760/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790950963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4203.013267] Lustre: 147545:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4206.307489] Lustre: Failing over lustre-MDT0000 [ 4206.324562] LustreError: 147543:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4206.539773] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4206.543581] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4206.616461] Lustre: server umount lustre-MDT0000 complete [ 4207.079327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4207.092240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.607773] Lustre: 147545:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790950952/real 1790950952] req@00000000f0613f11 x1877948063846784/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790950968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4208.643252] Lustre: 147545:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4210.022645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.051193] Lustre: Skipped 1 previous similar message [ 4211.589228] LustreError: 149339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4211.615633] LustreError: 149339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4215.270619] LustreError: 149338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4215.293246] LustreError: 149338:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4216.715089] LustreError: 149339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.385191] LustreError: 149343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.416171] LustreError: 149343:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4225.507773] LustreError: 149340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.544048] LustreError: 149340:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4225.780230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.929195] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4226.144556] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4226.209649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.270680] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4226.950832] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4231.128099] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4231.663624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4231.692642] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4231.725916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4241.895757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.898414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4241.910961] Lustre: Skipped 1 previous similar message [ 4241.926540] Lustre: Skipped 1 previous similar message [ 4245.407753] Lustre: server umount lustre-OST0000 complete [ 4252.136686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.142104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4252.151178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4257.248116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4257.257069] Lustre: Skipped 2 previous similar messages [ 4262.374150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4262.378942] Lustre: Skipped 1 previous similar message [ 4263.391467] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4263.536582] Lustre: server umount lustre-MDT0000 complete [ 4266.823214] LustreError: 151298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951026 with bad export cookie 3324031346035941001 [ 4266.826728] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.837152] LustreError: 151298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4267.214214] Lustre: server umount lustre-MDT0001 complete [ 4277.272577] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4280.211105] Key type lgssc unregistered [ 4280.491481] LNet: 154954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4280.502919] LNetError: 154954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4280.520809] LNet: Removed LNI 192.168.206.108@tcp [ 4281.331975] Key type .llcrypt unregistered [ 4281.333807] Key type ._llcrypt unregistered [ 4315.508899] Key type ._llcrypt registered [ 4315.513762] Key type .llcrypt registered [ 4315.634317] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4316.851918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4316.875955] alg: No test for adler32 (adler32-zlib) [ 4317.932772] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 4318.141552] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4319.839326] Key type lgssc registered [ 4321.017423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4328.884544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4328.903361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.409759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.582882] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4340.710135] LustreError: 156491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4340.729750] LustreError: 156491:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4344.146408] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4345.832092] LustreError: 156490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.357526] LustreError: 156519:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4348.368685] LustreError: 156519:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4348.386581] LustreError: 156519:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4348.767302] Lustre: server umount lustre-MDT0000 complete [ 4365.052271] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 4376.353489] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4416.174813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.400338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4416.420989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4416.520323] Lustre: lustre-MDT0000: new disk, initializing [ 4416.648052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.664759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4421.397607] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4431.480246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4431.688696] Lustre: 159825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4431.759722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4431.764521] Lustre: Skipped 1 previous similar message [ 4431.854442] Lustre: lustre-MDT0001: new disk, initializing [ 4431.980415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4432.027753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4432.043920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4437.535239] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4447.561435] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4453.459408] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4460.734734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4461.009901] Lustre: lustre-OST0000: new disk, initializing [ 4461.013599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4461.027622] Lustre: 161794:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4461.134181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.755465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4466.769194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4466.910445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4467.334601] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4477.387176] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4482.761321] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4484.375942] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:27:23 (1790951243) [ 4743.648623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.655688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4743.665851] Lustre: Skipped 2 previous similar messages [ 4743.681553] Lustre: Skipped 1 previous similar message [ 4748.305591] Lustre: server umount lustre-MDT0000 complete [ 4748.770645] LustreError: 159830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.791548] LustreError: 159830:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4751.890525] LustreError: 159817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951511 with bad export cookie 17351951615639745046 [ 4751.895743] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4751.908939] LustreError: 159817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4753.888114] LustreError: 159832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.888637] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.920915] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4756.897929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4758.196436] Lustre: server umount lustre-MDT0001 complete [ 4768.334962] Lustre: server umount lustre-OST0000 complete [ 4779.192082] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 4790.824762] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4831.729676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.102243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4832.136163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4832.219670] Lustre: lustre-MDT0000: new disk, initializing [ 4832.271405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4832.280812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4837.655291] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4848.864249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4849.060815] Lustre: 166465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4849.116148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4849.133418] Lustre: Skipped 1 previous similar message [ 4849.193974] Lustre: lustre-MDT0001: new disk, initializing [ 4849.278381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4849.319388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4849.340736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4855.117578] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4866.935130] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4873.861818] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4881.058489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4881.267908] Lustre: lustre-OST0000: new disk, initializing [ 4881.271289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4881.277973] Lustre: 168432:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4881.365093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.162044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4887.172732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4887.337795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4888.206836] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4897.949597] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4902.452811] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4902.648926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4906.520078] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4906.692806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4913.120431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4913.128495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4913.136821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4913.166209] Lustre: Skipped 1 previous similar message [ 4914.671390] Lustre: server umount lustre-OST0000 complete [ 4920.803781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4920.822677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.838306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4920.849301] Lustre: Skipped 1 previous similar message [ 4925.784463] Lustre: server umount lustre-MDT0000 complete [ 4928.523085] LustreError: 166472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4928.542346] LustreError: 166472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4930.120252] LustreError: 168431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951690 with bad export cookie 17351951615639903876 [ 4930.122166] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4930.146254] LustreError: 168431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4930.744421] Lustre: server umount lustre-MDT0001 complete [ 4941.386655] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:34:59 (1790951699) [ 4960.272844] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 4971.450877] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5005.906182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.143261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5006.204587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5006.329646] Lustre: lustre-MDT0000: new disk, initializing [ 5006.425839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5006.440933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5010.625026] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5021.384844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.637171] Lustre: 173187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5021.907146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5026.796547] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5035.636861] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5042.128622] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5049.406878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5049.700582] Lustre: lustre-OST0000: new disk, initializing [ 5049.703942] Lustre: Skipped 1 previous similar message [ 5049.709742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5049.715179] Lustre: Skipped 2 previous similar messages [ 5049.720805] Lustre: 175167:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5055.675436] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5057.567154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5057.573412] Lustre: Skipped 1 previous similar message [ 5057.584538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5057.664318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5063.443788] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5072.877368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.877379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5072.903274] Lustre: Skipped 1 previous similar message [ 5072.909137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5072.909144] Lustre: Skipped 2 previous similar messages [ 5072.948378] LustreError: Skipped 1 previous similar message [ 5076.142904] Lustre: server umount lustre-OST0000 complete [ 5083.107383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.133530] Lustre: Skipped 2 previous similar messages [ 5093.345914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5093.354390] Lustre: Skipped 6 previous similar messages [ 5094.368289] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5094.544845] Lustre: server umount lustre-MDT0000 complete [ 5098.025145] LustreError: 175166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951858 with bad export cookie 17351951615639904758 [ 5098.033262] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5098.035296] LustreError: 175166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5098.380903] Lustre: server umount lustre-MDT0001 complete [ 5106.725595] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:37:45 (1790951865) [ 5122.474913] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5133.156048] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5167.980910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.242355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5168.272308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5168.335701] Lustre: lustre-MDT0000: new disk, initializing [ 5168.403669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.412582] Lustre: Skipped 2 previous similar messages [ 5168.440980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5174.153337] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5185.423570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5185.496754] Lustre: 179495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5185.710577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5190.180689] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5199.925789] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5206.830779] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5214.639460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5215.037567] Lustre: 181473:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5221.425733] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5222.503665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5222.653754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5230.949815] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5243.366132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5243.370429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5243.375978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5243.394158] Lustre: Skipped 1 previous similar message [ 5244.638262] Lustre: server umount lustre-OST0000 complete [ 5254.623293] Lustre: server umount lustre-MDT0000 complete [ 5257.602969] LustreError: 179488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952017 with bad export cookie 17351951615639905836 [ 5257.607111] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5257.611471] LustreError: 179488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5257.856803] Lustre: server umount lustre-MDT0001 complete [ 5267.688689] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5269.884873] Key type lgssc unregistered [ 5270.107377] LNet: 183326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5270.116727] LNetError: 183326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5270.138620] LNet: Removed LNI 192.168.206.108@tcp [ 5270.778109] Key type .llcrypt unregistered [ 5270.780374] Key type ._llcrypt unregistered [ 5293.352668] Key type ._llcrypt registered [ 5293.355725] Key type .llcrypt registered [ 5293.474485] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5294.491618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5294.510649] alg: No test for adler32 (adler32-zlib) [ 5295.460388] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 5295.672605] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5297.335203] Key type lgssc registered [ 5298.348734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5307.457231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5307.476447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.866493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5314.141754] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5318.118170] LustreError: 184766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5318.130318] LustreError: 184766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5323.233381] LustreError: 184765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.498268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.893643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5328.774836] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5337.071615] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5342.700347] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5349.789311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5350.097024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5353.226470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5356.695783] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5366.562797] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5389.281521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5389.293174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.298209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5389.304176] Lustre: Skipped 1 previous similar message [ 5389.318473] Lustre: Skipped 1 previous similar message [ 5390.533305] Lustre: server umount lustre-OST0000 complete [ 5394.400677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.411269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5394.412266] Lustre: Skipped 1 previous similar message [ 5394.424747] Lustre: Skipped 1 previous similar message [ 5399.521459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5399.529227] Lustre: Skipped 1 previous similar message [ 5404.640117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5404.650822] Lustre: Skipped 1 previous similar message [ 5408.223302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5408.437345] Lustre: server umount lustre-MDT0000 complete [ 5409.762265] LustreError: 184765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.780704] LustreError: 184765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5412.015187] LustreError: 184746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952172 with bad export cookie 14946169253627081532 [ 5412.016204] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5412.033325] LustreError: 184746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5412.533629] Lustre: server umount lustre-MDT0001 complete [ 5424.175707] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5427.665913] Key type lgssc unregistered [ 5427.964603] LNet: 188325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5427.975839] LNetError: 188325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5427.989814] LNet: Removed LNI 192.168.206.108@tcp [ 5428.835722] Key type .llcrypt unregistered [ 5428.837876] Key type ._llcrypt unregistered [ 5454.327775] Key type ._llcrypt registered [ 5454.330849] Key type .llcrypt registered [ 5454.413819] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5455.541742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5455.565090] alg: No test for adler32 (adler32-zlib) [ 5456.880287] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 5457.132796] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5458.807688] Key type lgssc registered [ 5460.215843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5469.789654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5469.829918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.408481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5476.324637] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5481.954990] LustreError: 189759:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5481.988546] LustreError: 189759:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5485.149351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.475031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5490.371498] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5499.544069] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5505.771418] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5513.005842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5513.485425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.641823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5519.987634] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5527.682785] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5538.275175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5538.285363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5538.308620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5539.750382] Lustre: lustre-OST0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 5541.348613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5541.349733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5541.359180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5544.835283] Lustre: lustre-OST0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 5544.841470] Lustre: Skipped 1 previous similar message [ 5549.957400] Lustre: lustre-OST0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 5549.962795] Lustre: Skipped 2 previous similar messages [ 5551.076516] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5551.256485] Lustre: server umount lustre-OST0000 complete [ 5558.290162] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff923f50dd3000.ost_server_uuid 50 [ 5559.745614] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5576.172772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.193496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5576.211355] Lustre: Skipped 1 previous similar message [ 5579.499796] Lustre: server umount lustre-MDT0000 complete [ 5581.280918] LustreError: 191001:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.316469] LustreError: 191001:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5582.738779] LustreError: 189740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952342 with bad export cookie 12564450987865012855 [ 5582.743407] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.753224] LustreError: 189740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5583.056914] Lustre: server umount lustre-MDT0001 complete [ 5592.792405] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:45:50 (1790952350) [ 5600.962199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5601.354496] LustreError: 193678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.415228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5606.387399] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5606.882151] LustreError: 193679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.989084] LustreError: 193678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.328938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5617.773583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5623.970087] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5633.278426] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5638.813748] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5645.056995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5645.421260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5646.448726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5650.822580] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5657.961776] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5665.209508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.422258] Lustre: lustre-OST0001: new disk, initializing [ 5665.426817] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5665.432236] Lustre: 196523:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5665.525174] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5671.834739] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5672.021497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5672.041254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5672.152887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5679.526325] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5682.658187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5682.669950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.685208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5682.699098] Lustre: Skipped 2 previous similar messages [ 5687.479846] Lustre: server umount lustre-OST0000 complete [ 5687.784817] LustreError: 196890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.806096] LustreError: 196890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5693.182782] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5693.460515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5697.893510] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5698.082041] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5705.603400] LustreError: 195504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5705.630170] LustreError: 195504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5713.387152] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5713.393765] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5717.949506] Lustre: server umount lustre-OST0001 complete [ 5723.617336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5723.633124] Lustre: Skipped 1 previous similar message [ 5723.638668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5723.646610] Lustre: Skipped 5 previous similar messages [ 5735.391529] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5735.668232] Lustre: server umount lustre-MDT0000 complete [ 5738.977171] LustreError: 194609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.022965] LustreError: 194609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5740.818614] LustreError: 193658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952500 with bad export cookie 12564450987865013926 [ 5740.826211] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5740.852560] LustreError: 193658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5741.349241] Lustre: server umount lustre-MDT0001 complete [ 5768.384551] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:48:46 (1790952526) [ 5776.460787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.818475] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5776.845695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5777.042517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5781.402610] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5789.779494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5789.837183] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5789.865864] Lustre: 199964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5789.942919] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5789.951922] Lustre: Skipped 4 previous similar messages [ 5793.577751] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5799.419381] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5804.449751] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5811.604507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5811.872782] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5817.904929] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5818.532696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5825.387572] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5832.539435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5832.652499] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5839.061910] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5840.035806] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5846.565460] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5849.056516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5849.062404] LustreError: Skipped 1 previous similar message [ 5849.065383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.076211] Lustre: Skipped 1 previous similar message [ 5855.214249] Lustre: server umount lustre-OST0000 complete [ 5855.712855] LustreError: 201811:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.742649] LustreError: 201811:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5870.564638] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5870.583196] LustreError: Skipped 1 previous similar message [ 5875.753930] Lustre: server umount lustre-OST0001 complete [ 5881.314121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.330030] Lustre: Skipped 4 previous similar messages [ 5885.932121] Lustre: server umount lustre-MDT0000 complete [ 5889.284331] LustreError: 199955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952649 with bad export cookie 12564450987865015333 [ 5889.301100] LustreError: 199955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5889.302858] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5889.732980] Lustre: server umount lustre-MDT0001 complete [ 5913.554348] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:51:12 (1790952672) [ 5928.113736] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5938.848353] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5979.593964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.869854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5979.926142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5980.029069] Lustre: lustre-MDT0000: new disk, initializing [ 5980.093667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5980.098524] Lustre: Skipped 3 previous similar messages [ 5980.117109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5983.502575] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5993.134476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5993.228399] Lustre: 208057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5993.270132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5993.273917] Lustre: Skipped 1 previous similar message [ 5993.375194] Lustre: lustre-MDT0001: new disk, initializing [ 5993.544411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5993.562575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5997.235972] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6005.766319] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6011.451799] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6018.110761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6018.412894] Lustre: lustre-OST0000: new disk, initializing [ 6018.421104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6018.433347] Lustre: 210027:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6023.983231] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6028.344194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6028.355323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6028.423277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6032.484447] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6037.043816] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6037.236353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6041.757501] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6042.027570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6044.130896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6044.132536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.136278] LustreError: Skipped 2 previous similar messages [ 6044.137248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6044.137252] Lustre: Skipped 10 previous similar messages [ 6044.192325] Lustre: Skipped 1 previous similar message [ 6050.381898] Lustre: server umount lustre-OST0000 complete [ 6060.524510] Lustre: server umount lustre-MDT0000 complete [ 6064.100870] LustreError: 208069:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6064.128649] LustreError: 208069:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6064.148483] LustreError: 208048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952824 with bad export cookie 12564450987865017118 [ 6064.149614] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6064.163140] LustreError: 208048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6064.560975] Lustre: server umount lustre-MDT0001 complete [ 6072.594551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.741093] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6083.923930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.087110] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6097.088492] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6103.586325] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6109.699770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6110.008147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6110.019307] Lustre: Skipped 4 previous similar messages [ 6115.844337] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6121.997556] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6126.229504] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6126.439411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6130.362541] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6130.584457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6133.729360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6133.736453] LustreError: Skipped 1 previous similar message [ 6138.480475] Lustre: server umount lustre-OST0000 complete [ 6144.360753] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6147.835323] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6175.147616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6180.597716] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6184.610724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.618330] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6186.248201] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6195.258426] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 6199.282986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.293975] Lustre: Skipped 4 previous similar messages [ 6200.483994] Lustre: server umount lustre-OST0000 complete [ 6209.693236] Lustre: server umount lustre-MDT0000 complete [ 6213.069790] LustreError: 212975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952973 with bad export cookie 12564450987865018000 [ 6213.076757] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6213.079519] LustreError: 212975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6213.411144] Lustre: server umount lustre-MDT0001 complete [ 6226.376885] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:56:24 (1790952984) [ 6236.789028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.548500] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6250.705118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.747483] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6263.310519] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6270.361595] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6278.627306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6285.127104] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6291.726155] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6295.692966] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6295.889939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6300.205770] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6300.397292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6306.599034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6306.787921] Lustre: lustre-OST0001: new disk, initializing [ 6306.791377] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6306.795609] Lustre: 221575:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6312.867157] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6316.147794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6316.169639] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6316.270052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6322.033588] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6326.269328] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6326.523953] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6331.305125] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6331.531939] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6337.010453] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6337.024727] Lustre: Skipped 16 previous similar messages [ 6339.595311] Lustre: server umount lustre-OST0001 complete [ 6342.116139] LustreError: 221863:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.153275] LustreError: 221863:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6347.578359] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6347.806623] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6352.040827] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6352.211829] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6383.713154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6384.092258] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6384.097669] Lustre: Skipped 5 previous similar messages [ 6389.228463] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6389.252337] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6389.253105] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6391.218151] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6399.356153] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6400.602549] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6404.595962] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6404.608487] LustreError: Skipped 2 previous similar messages [ 6408.470985] Lustre: server umount lustre-OST0001 complete [ 6438.367117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6438.480231] Lustre: server umount lustre-OST0000 complete [ 6456.287220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6456.460549] Lustre: server umount lustre-MDT0000 complete [ 6459.917462] LustreError: 218186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953219 with bad export cookie 12564450987865018959 [ 6459.926100] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6466.348815] Lustre: server umount lustre-MDT0001 complete [ 6490.940921] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:00:49 (1790953249) [ 6496.815926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.159883] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6497.201806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6501.488945] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6509.138350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.213607] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6509.238401] Lustre: 226819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6512.645407] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6518.644620] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6523.769782] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6530.539740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6530.686374] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6536.318950] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6543.148950] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6552.046510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6552.166359] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6558.541760] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6564.603754] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6566.010106] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6570.165419] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6570.353613] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6573.739206] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6573.920222] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6578.863262] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff923f50982000.ost_server_uuid 50 [ 6580.374327] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff923f50982000.ost_server_uuid in IDLE state after 0 sec [ 6581.112539] Lustre: Permanently deactivating lustre-OST0001 [ 6581.128853] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6583.681781] Lustre: Permanently reactivating lustre-OST0001 [ 6589.411937] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6589.415829] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.422090] Lustre: Skipped 10 previous similar messages [ 6592.540041] Lustre: server umount lustre-OST0001 complete [ 6601.585079] Lustre: server umount lustre-OST0000 complete [ 6611.625605] Lustre: server umount lustre-MDT0000 complete [ 6616.166868] LustreError: 226812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953376 with bad export cookie 12564450987865020247 [ 6616.175035] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6616.176995] LustreError: 226812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6616.574450] Lustre: server umount lustre-MDT0001 complete [ 6639.208587] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:03:18 (1790953398) [ 6645.338122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.548429] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6645.575751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6645.585853] Lustre: Skipped 5 previous similar messages [ 6649.480181] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6656.927105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6657.044650] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6657.068395] Lustre: 233521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6660.804622] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6667.108393] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6673.159639] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6679.089737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6679.312082] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6679.342095] Lustre: Permanently deactivating lustre-OST0000 [ 6679.351884] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6684.689800] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6685.634731] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6693.668580] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6701.456487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6701.533469] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6708.234301] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6711.500263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6717.139615] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6719.095895] Lustre: Permanently reactivating lustre-OST0000 [ 6719.110785] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6719.118857] Lustre: Skipped 2 previous similar messages [ 6719.606227] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6719.607649] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000138e47fd, cur 1790953479 deadline 1790953474 last 1790953444 [ 6719.618720] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6719.645967] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6719.657634] Lustre: Skipped 1 previous similar message [ 6719.703857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6741.893841] Lustre: server umount lustre-OST0001 complete [ 6752.415773] Lustre: server umount lustre-OST0000 complete [ 6762.740398] Lustre: server umount lustre-MDT0000 complete [ 6766.541878] LustreError: 233513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953526 with bad export cookie 12564450987865021892 [ 6766.551132] LustreError: 233513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6766.554730] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6767.049780] Lustre: server umount lustre-MDT0001 complete [ 6778.055036] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6780.572317] Key type lgssc unregistered [ 6780.859826] LNet: 238399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6780.869484] LNetError: 238399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6780.894179] LNet: Removed LNI 192.168.206.108@tcp [ 6782.013966] Key type .llcrypt unregistered [ 6782.018065] Key type ._llcrypt unregistered [ 6804.934527] Key type ._llcrypt registered [ 6804.940496] Key type .llcrypt registered [ 6805.023826] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6805.966062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6805.992908] alg: No test for adler32 (adler32-zlib) [ 6806.943597] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 6807.194466] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6808.839638] Key type lgssc registered [ 6809.911682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6820.290984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6820.375945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6822.214292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6827.128430] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6832.610429] LustreError: 239882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6832.630791] LustreError: 239882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6837.733551] LustreError: 239883:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6839.224782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.338959] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6839.363921] Lustre: 239872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6839.374592] Lustre: Permanently deactivating lustre-MDT0001 [ 6839.385302] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6839.685977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6843.855233] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6845.989421] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6852.165781] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6861.120044] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6869.709029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6870.258279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6876.924152] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6884.623280] LustreError: 241726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.636623] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6884.926764] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6889.958140] LustreError: 241726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6891.858056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6892.266396] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6893.300030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6893.330051] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6893.356618] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6898.376361] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6905.699485] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6908.216823] Lustre: Permanently reactivating lustre-MDT0001 [ 6908.228578] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6908.237445] Lustre: Skipped 2 previous similar messages [ 6912.920873] Lustre: lustre-MDT0001: Client ff231489-3ac1-4d96-97d4-278c7d95689c (at 192.168.206.8@tcp) reconnecting [ 6914.662899] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6914.678054] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6914.691052] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6914.701593] LustreError: 243453:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6914.703634] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6914.708332] LustreError: 243453:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6914.708437] LustreError: 243453:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.108@tcp: cfg command failed: rc = -108 [ 6914.708459] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6914.708459] [ 6914.709676] LustreError: 239874:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6919.089322] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6920.481619] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6926.212744] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6926.357759] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6931.062147] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6931.300114] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6933.953142] Lustre: Permanently deactivating lustre-MDT0001 [ 6933.967811] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6933.973302] Lustre: Skipped 2 previous similar messages [ 6941.220884] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6945.761375] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6945.763243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6945.774313] Lustre: Skipped 1 previous similar message [ 6950.884144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6950.898945] Lustre: Skipped 6 previous similar messages [ 6951.985101] Lustre: server umount lustre-MDT0000 complete [ 6955.480224] LustreError: 239864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953715 with bad export cookie 6052445022259022695 [ 6955.483771] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6955.488129] LustreError: 239864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6955.692872] Lustre: server umount lustre-MDT0001 complete [ 6964.980530] Lustre: server umount lustre-OST0000 complete [ 6974.478613] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:08:53 (1790953733) [ 6983.979949] Lustre: server umount lustre-OST0001 complete [ 6990.647813] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7001.022341] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7031.222612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.427723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7031.439382] Lustre: Skipped 2 previous similar messages [ 7031.475698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7031.537923] Lustre: lustre-MDT0000: new disk, initializing [ 7031.639132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7031.666513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7035.947201] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7047.302757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7047.413584] Lustre: 248142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7047.454443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7047.457832] Lustre: Skipped 1 previous similar message [ 7047.511930] Lustre: lustre-MDT0001: new disk, initializing [ 7047.586911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7047.623629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7047.630114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7052.145648] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7059.770200] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7064.388628] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7070.916876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7071.177771] Lustre: lustre-OST0000: new disk, initializing [ 7071.180957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7071.187773] Lustre: 250111:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7071.296508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7076.965261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7076.989104] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7077.093902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7078.176415] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7086.625964] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7088.683130] LustreError: 248150:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7096.309979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7096.472549] Lustre: lustre-OST0001: new disk, initializing [ 7096.482562] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7096.491907] Lustre: 251226:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7098.683365] LustreError: 248150:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7098.705954] LustreError: 250979:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7101.919914] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7105.130703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7105.143811] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7105.223419] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7108.783160] LustreError: 250979:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7111.010294] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7112.764915] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7115.166811] Lustre: lustre-OST0001: Not available for connect from 192.168.206.8@tcp (stopping) [ 7115.750032] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7115.752517] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7115.768043] LustreError: Skipped 1 previous similar message [ 7115.803521] Lustre: Skipped 3 previous similar messages [ 7120.258837] Lustre: lustre-OST0001: Not available for connect from 192.168.206.8@tcp (stopping) [ 7120.264954] Lustre: Skipped 2 previous similar messages [ 7120.828544] Lustre: server umount lustre-OST0001 complete [ 7120.868125] LustreError: 250096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7120.885135] LustreError: 250096:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7131.113352] LustreError: 250094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7131.148529] LustreError: 250094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7138.787448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.803202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.822117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7142.665650] Lustre: server umount lustre-OST0000 complete [ 7146.471625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7146.480947] Lustre: Skipped 2 previous similar messages [ 7151.588368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7151.602764] Lustre: Skipped 6 previous similar messages [ 7152.039848] Lustre: server umount lustre-MDT0000 complete [ 7155.696378] LustreError: 248135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953915 with bad export cookie 6052445022259025509 [ 7155.698406] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7155.703536] LustreError: 248135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7170.018654] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7170.335919] Lustre: server umount lustre-MDT0001 complete [ 7179.503888] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7182.073688] Key type lgssc unregistered [ 7182.361518] LNet: 253258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7182.385926] LNetError: 253258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7182.424302] LNet: Removed LNI 192.168.206.108@tcp [ 7183.490738] Key type .llcrypt unregistered [ 7183.494545] Key type ._llcrypt unregistered [ 7226.624892] Key type ._llcrypt registered [ 7226.628276] Key type .llcrypt registered [ 7226.714218] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7228.020503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7228.037979] alg: No test for adler32 (adler32-zlib) [ 7229.068651] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 7229.293290] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7230.991606] Key type lgssc registered [ 7232.182372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7239.311669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7239.327615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.562364] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7240.581603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7240.741159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7244.841279] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7253.089689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.159785] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7253.181779] Lustre: 255483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7253.363385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7256.974866] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7263.588926] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7269.356698] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7276.023436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7276.187230] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7276.370050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7282.341722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7282.639959] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7290.363892] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7344.096193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7344.103468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.116896] LustreError: Skipped 1 previous similar message [ 7344.127169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7344.134130] Lustre: Skipped 1 previous similar message [ 7344.158118] Lustre: Skipped 1 previous similar message [ 7348.018561] Lustre: server umount lustre-OST0000 complete [ 7352.473292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7366.031098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.350896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.647193] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7380.068721] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7384.748297] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7384.782366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7384.787533] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7384.806965] Lustre: Skipped 1 previous similar message [ 7420.898349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7420.909738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7420.921131] Lustre: Skipped 1 previous similar message [ 7420.938665] Lustre: Skipped 1 previous similar message [ 7425.185088] Lustre: server umount lustre-OST0000 complete [ 7431.143793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7431.161140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7431.169391] Lustre: Skipped 1 previous similar message [ 7434.914280] Lustre: server umount lustre-MDT0000 complete [ 7438.774449] LustreError: 255477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954198 with bad export cookie 7780421213175376653 [ 7438.780049] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7438.792959] LustreError: 255477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7439.227526] Lustre: server umount lustre-MDT0001 complete [ 7449.138461] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7451.566889] Key type lgssc unregistered [ 7451.817873] LNet: 260905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7451.830230] LNetError: 260905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7451.850146] LNet: Removed LNI 192.168.206.108@tcp [ 7452.664252] Key type .llcrypt unregistered [ 7452.668667] Key type ._llcrypt unregistered [ 7473.759096] Key type ._llcrypt registered [ 7473.760996] Key type .llcrypt registered [ 7473.866439] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7475.092988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7475.136924] alg: No test for adler32 (adler32-zlib) [ 7476.238579] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 7476.515567] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7478.199258] Key type lgssc registered [ 7479.215943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7486.770708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7486.788717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7488.100372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.179498] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7498.723936] LustreError: 262315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7498.743778] LustreError: 262315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7499.487423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.714620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7503.143642] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7510.130523] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7516.126268] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7523.614162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.107188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7530.759310] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7533.364405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7537.431399] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7561.188777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7561.193815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.206080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7564.261295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7564.264479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7564.266641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.902817] Lustre: server umount lustre-OST0000 complete [ 7569.383243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7569.406310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7569.411525] Lustre: Skipped 2 previous similar messages [ 7574.498640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7574.507984] Lustre: Skipped 1 previous similar message [ 7575.308888] Lustre: server umount lustre-MDT0000 complete [ 7578.512039] LustreError: 262297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954338 with bad export cookie 701917553726497465 [ 7578.523155] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7578.524434] LustreError: 262297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7578.804094] Lustre: server umount lustre-MDT0001 complete [ 7588.995400] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7591.479908] Key type lgssc unregistered [ 7591.780764] LNet: 266620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7591.801515] LNetError: 266620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7591.815904] LNet: Removed LNI 192.168.206.108@tcp [ 7592.551588] Key type .llcrypt unregistered [ 7592.554828] Key type ._llcrypt unregistered [ 7607.968329] Key type ._llcrypt registered [ 7607.970883] Key type .llcrypt registered [ 7608.055421] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7608.788030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7608.855632] alg: No test for adler32 (adler32-zlib) [ 7609.821647] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 7610.030708] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7611.687285] Key type lgssc registered [ 7612.670534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7619.399177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7619.424750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.806941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7624.927252] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7630.311066] LustreError: 267858:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.340096] LustreError: 267858:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7633.970472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.334210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7638.502436] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7645.558978] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7650.487136] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7656.561748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7656.888993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.860817] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7665.093360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7669.651652] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7685.600271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7685.602093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7685.612193] LustreError: Skipped 1 previous similar message [ 7685.614348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7685.668072] Lustre: Skipped 1 previous similar message [ 7689.193169] Lustre: server umount lustre-OST0000 complete [ 7695.840162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7695.841219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7695.849526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7695.849533] Lustre: Skipped 1 previous similar message [ 7695.888764] Lustre: Skipped 1 previous similar message [ 7699.174640] Lustre: server umount lustre-MDT0000 complete [ 7700.965594] LustreError: 267857:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7700.977374] LustreError: 267857:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7702.927567] LustreError: 267837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954462 with bad export cookie 13698157570971861688 [ 7702.930834] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7702.941089] LustreError: 267837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7703.416627] Lustre: server umount lustre-MDT0001 complete [ 7714.199435] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7716.892144] Key type lgssc unregistered [ 7717.198653] LNet: 271533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7717.208570] LNetError: 271533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7717.228856] LNet: Removed LNI 192.168.206.108@tcp [ 7717.980617] Key type .llcrypt unregistered [ 7717.983643] Key type ._llcrypt unregistered [ 7744.519180] Key type ._llcrypt registered [ 7744.521966] Key type .llcrypt registered [ 7744.644627] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7745.851462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7745.992552] alg: No test for adler32 (adler32-zlib) [ 7747.177746] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 7747.537316] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7749.223312] Key type lgssc registered [ 7750.320862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7758.424853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7758.447783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7760.001462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7764.431306] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7770.596146] LustreError: 272965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.620629] LustreError: 272965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7773.290440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7773.663780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7777.927485] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7784.595502] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7789.516768] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7794.808499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7795.050925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.413242] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7806.274781] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7806.323260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7823.847928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7823.859116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.871271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.916902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7826.917074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.924190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.934369] Lustre: Skipped 1 previous similar message [ 7829.115752] Lustre: server umount lustre-OST0000 complete [ 7835.105101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7835.113735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.140344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.153542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.156206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7838.957841] Lustre: server umount lustre-MDT0000 complete [ 7842.272763] LustreError: 274013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7842.301638] LustreError: 274013:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7843.038251] LustreError: 272946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954603 with bad export cookie 5539949949322528915 [ 7843.071392] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7843.404417] Lustre: server umount lustre-MDT0001 complete [ 7854.906734] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7857.394912] Key type lgssc unregistered [ 7857.576472] LNet: 277269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7857.583785] LNetError: 277269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7857.599151] LNet: Removed LNI 192.168.206.108@tcp [ 7858.271618] Key type .llcrypt unregistered [ 7858.273785] Key type ._llcrypt unregistered [ 7876.794156] Key type ._llcrypt registered [ 7876.795677] Key type .llcrypt registered [ 7876.956403] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7879.031575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7879.128627] alg: No test for adler32 (adler32-zlib) [ 7880.317835] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 7880.608746] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7882.287765] Key type lgssc registered [ 7883.766594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7892.681142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7892.694452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7894.182727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.423349] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7904.738826] LustreError: 278520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7904.765347] LustreError: 278520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7907.233266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7907.549922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7910.431670] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7916.644979] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7921.171844] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7927.715171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.954183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7933.110910] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7939.247056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7939.648119] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7949.822339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7949.831678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7949.838309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7949.844020] Lustre: Skipped 1 previous similar message [ 7949.859084] Lustre: Skipped 1 previous similar message [ 7954.921018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7954.937530] Lustre: Skipped 1 previous similar message [ 7955.233921] Lustre: server umount lustre-OST0000 complete [ 7960.034352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.044847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7960.059616] Lustre: Skipped 1 previous similar message [ 7960.079271] Lustre: Skipped 1 previous similar message [ 7965.163711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7965.167967] Lustre: Skipped 1 previous similar message [ 7965.494723] Lustre: server umount lustre-MDT0000 complete [ 7968.881646] LustreError: 278488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954728 with bad export cookie 10668110631462354681 [ 7968.889727] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7968.897300] LustreError: 278488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7969.173152] Lustre: server umount lustre-MDT0001 complete [ 7979.142133] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7981.410549] Key type lgssc unregistered [ 7981.698865] LNet: 282219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7981.710420] LNetError: 282219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7981.760977] LNet: Removed LNI 192.168.206.108@tcp [ 7982.563305] Key type .llcrypt unregistered [ 7982.566683] Key type ._llcrypt unregistered [ 7994.209551] Key type ._llcrypt registered [ 7994.212859] Key type .llcrypt registered [ 7994.360736] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8012.237672] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8027.593023] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8028.128623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8028.190881] alg: No test for adler32 (adler32-zlib) [ 8029.164502] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 8029.359860] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8031.058525] Key type lgssc registered [ 8032.091172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8064.314419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8064.347317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.598746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8065.616301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8065.682575] Lustre: lustre-MDT0000: new disk, initializing [ 8065.763924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8065.775947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8069.170722] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8077.865130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8078.023718] Lustre: 285788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8078.097882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8078.111732] Lustre: Skipped 1 previous similar message [ 8078.177384] Lustre: lustre-MDT0001: new disk, initializing [ 8078.242730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8078.284817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8078.294931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8081.756781] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8090.615241] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8095.645353] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8103.749269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8104.202402] Lustre: lustre-OST0000: new disk, initializing [ 8104.209676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8104.217609] Lustre: 287759:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8104.451894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8110.280768] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8112.692088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8112.702677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8112.797636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8118.470967] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8123.638854] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8123.932195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8128.846670] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8129.238849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8133.088517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8133.107580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.124191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8133.602846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8133.616696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.635136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8137.956690] Lustre: server umount lustre-OST0000 complete [ 8143.841436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8143.850266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8143.853707] Lustre: Skipped 1 previous similar message [ 8148.868531] Lustre: server umount lustre-MDT0000 complete [ 8152.863491] LustreError: 285779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954912 with bad export cookie 9537161822481629707 [ 8152.864512] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8152.881736] LustreError: 285779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8153.147303] Lustre: server umount lustre-MDT0001 complete [ 8163.784125] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:28:42 (1790954922) [ 8171.573383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.084463] LustreError: 290169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8172.102384] LustreError: 290169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8172.160209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8176.983412] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8177.632385] LustreError: 290170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.755361] LustreError: 290169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8184.768586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.246553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8191.055712] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8200.505628] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8208.443301] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8214.382973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8214.595847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.071460] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8224.791123] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8238.049761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8238.056851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.071415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.077368] Lustre: Skipped 1 previous similar message [ 8239.117440] Lustre: server umount lustre-OST0000 complete [ 8246.241728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.244252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.252495] Lustre: Skipped 1 previous similar message [ 8249.119853] Lustre: server umount lustre-MDT0000 complete [ 8251.360773] LustreError: 290164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.380611] LustreError: 290164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8252.963023] LustreError: 290150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955013 with bad export cookie 9537161822481630589 [ 8252.964344] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8252.972928] LustreError: 290150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8263.299218] Lustre: server umount lustre-MDT0001 complete [ 8271.113316] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8272.738411] Key type lgssc unregistered [ 8272.980585] LNet: 293725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8272.988465] LNetError: 293725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8273.002177] LNet: Removed LNI 192.168.206.108@tcp [ 8273.617343] Key type .llcrypt unregistered [ 8273.618697] Key type ._llcrypt unregistered [ 8297.129248] Key type ._llcrypt registered [ 8297.131683] Key type .llcrypt registered [ 8297.201866] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8312.048718] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8312.792594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8312.826092] alg: No test for adler32 (adler32-zlib) [ 8313.895963] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 8314.132685] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8315.855278] Key type lgssc registered [ 8317.275787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8356.854863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8356.875302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8358.013533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8358.030548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8358.093952] Lustre: lustre-MDT0000: new disk, initializing [ 8358.194284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8358.208813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8362.655649] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8372.471601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8372.673948] Lustre: 297583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8372.726952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8372.730735] Lustre: Skipped 1 previous similar message [ 8372.793847] Lustre: lustre-MDT0001: new disk, initializing [ 8372.883278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8372.912755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8372.926989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8377.274036] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8386.459928] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8392.069307] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8398.046498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8398.299514] Lustre: lustre-OST2710: new disk, initializing [ 8398.303686] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8398.311312] Lustre: 299550:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8398.435587] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8403.083613] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8408.619890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8408.640858] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8408.778119] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8410.813879] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8416.594155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8416.738915] Lustre: lustre-OST03e8: new disk, initializing [ 8416.745682] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8416.750691] Lustre: 300613:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8416.847406] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8422.338194] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8422.932587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8422.937696] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8423.029418] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8428.538322] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8434.706443] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8434.860391] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8437.924665] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8438.033268] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8441.218381] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8441.402327] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8446.107727] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8446.290609] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8450.020220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8450.031215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8450.049323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.228532] Lustre: server umount lustre-MDT0000 complete [ 8453.603422] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8453.609581] Lustre: Skipped 1 previous similar message [ 8455.007084] LustreError: 297575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955215 with bad export cookie 17927518843499747500 [ 8455.011388] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8455.014931] LustreError: 297575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8455.328502] Lustre: server umount lustre-MDT0001 complete [ 8469.396439] Lustre: server umount lustre-OST2710 complete [ 8483.114700] Lustre: server umount lustre-OST03e8 complete [ 8497.247379] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8510.361270] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8544.659344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8544.884674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8544.902198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8544.965192] Lustre: lustre-MDT0000: new disk, initializing [ 8545.023439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8545.036398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8550.702709] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8561.132153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8561.237637] Lustre: 305861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8561.308078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8561.316352] Lustre: Skipped 1 previous similar message [ 8561.403754] Lustre: lustre-MDT0001: new disk, initializing [ 8561.520307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8561.558144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8561.575213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8565.070856] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8571.800369] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8575.720728] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8579.949882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8580.088426] Lustre: lustre-OST0000: new disk, initializing [ 8580.091707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8580.097603] Lustre: 307829:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8580.184262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8584.783710] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8589.334871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8589.339216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8589.400241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8591.456381] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8594.802602] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8594.959520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8598.064231] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8598.214958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8604.646624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8604.653750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.658159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8604.672258] Lustre: Skipped 2 previous similar messages [ 8605.717176] Lustre: server umount lustre-OST0000 complete [ 8609.761827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8609.762811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8609.770427] Lustre: Skipped 1 previous similar message [ 8609.774891] Lustre: Skipped 1 previous similar message [ 8614.641183] Lustre: server umount lustre-MDT0000 complete [ 8614.880815] LustreError: 305874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8614.896715] LustreError: 305874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 8617.139557] LustreError: 305854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955377 with bad export cookie 17927518843499749180 [ 8617.141721] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8617.153050] LustreError: 305854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8617.426575] Lustre: server umount lustre-MDT0001 complete [ 8623.966792] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:36:22 (1790955382) [ 8625.098480] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8626.585517] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:36:25 (1790955385) [ 8629.294896] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8648.340453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.544737] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8648.572676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8648.678060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8651.822577] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8658.329857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8658.437811] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8658.465284] Lustre: 311685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8662.609827] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8682.987290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8683.089737] Lustre: Denying initial registration attempt from nid 192.168.206.108@tcp, specified as failover [ 8683.097153] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8683.102466] LustreError: 313138:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 8683.106930] LustreError: 313138:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8683.110312] LustreError: 313138:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8683.149943] Lustre: server umount lustre-OST0000 complete [ 8683.153455] LustreError: 313138:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8689.633173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.634230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8689.637310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8689.637318] Lustre: Skipped 1 previous similar message [ 8689.643751] Lustre: Skipped 1 previous similar message [ 8689.660264] LustreError: Skipped 1 previous similar message [ 8694.752228] LustreError: 311697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8694.770719] LustreError: 311697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8695.831159] Lustre: server umount lustre-MDT0000 complete [ 8698.829449] LustreError: 311678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955458 with bad export cookie 17927518843499750062 [ 8698.834519] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8698.838989] LustreError: 311678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8699.071435] Lustre: server umount lustre-MDT0001 complete [ 8708.760381] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8710.695755] Key type lgssc unregistered [ 8710.861304] LNet: 314258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8710.865704] LNetError: 314258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8710.878670] LNet: Removed LNI 192.168.206.108@tcp [ 8711.499700] Key type .llcrypt unregistered [ 8711.501691] Key type ._llcrypt unregistered [ 8720.378859] Key type ._llcrypt registered [ 8720.380749] Key type .llcrypt registered [ 8720.456946] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8721.113901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8721.186078] alg: No test for adler32 (adler32-zlib) [ 8722.134585] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 8722.339676] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8723.983455] Key type lgssc registered [ 8724.832819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8760.745824] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8766.654640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8766.675387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8767.886817] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8767.919958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8768.039547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.939623] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8776.834196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8776.910890] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8776.934942] Lustre: 317014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8777.116197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.140218] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8785.262174] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8788.480364] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8805.801315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8805.939237] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8806.101689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8811.036203] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8816.752182] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8820.710394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8820.712037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8820.717054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8820.728740] Lustre: Skipped 1 previous similar message [ 8820.738472] Lustre: Skipped 1 previous similar message [ 8824.461417] Lustre: server umount lustre-OST0000 complete [ 8828.384785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8828.402561] LustreError: Skipped 1 previous similar message [ 8828.416685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.443182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8830.944943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.945260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8830.973751] Lustre: Skipped 1 previous similar message [ 8833.516489] Lustre: server umount lustre-MDT0000 complete [ 8836.613464] LustreError: 317007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955596 with bad export cookie 12028764665027323147 [ 8836.619334] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8836.624446] LustreError: 317007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8836.926674] Lustre: server umount lustre-MDT0001 complete [ 8845.787717] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8847.974995] Key type lgssc unregistered [ 8848.202585] LNet: 320585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8848.213135] LNetError: 320585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8848.239536] LNet: Removed LNI 192.168.206.108@tcp [ 8848.956485] Key type .llcrypt unregistered [ 8848.958370] Key type ._llcrypt unregistered [ 8866.722945] Key type ._llcrypt registered [ 8866.726295] Key type .llcrypt registered [ 8866.794412] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8867.556969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8867.578111] alg: No test for adler32 (adler32-zlib) [ 8868.576722] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 8868.773848] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8870.415578] Key type lgssc registered [ 8871.338951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8876.388246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8876.417439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8877.689694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.405979] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8885.724033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8885.954919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8888.709602] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8893.414699] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8896.592670] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8902.695941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.897620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8906.743951] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8911.386136] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8932.320096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8932.324276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.336404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8932.835230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.842098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8932.846323] Lustre: Skipped 1 previous similar message [ 8932.853508] Lustre: Skipped 1 previous similar message [ 8934.878302] Lustre: server umount lustre-MDT0000 complete [ 8937.843154] LustreError: 321977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955697 with bad export cookie 5679164796917668754 [ 8937.846187] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8937.858032] LustreError: 321977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8937.952861] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8937.953952] LustreError: 324485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.970633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8937.985442] LustreError: 324485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8937.987512] Lustre: Skipped 1 previous similar message [ 8943.080182] LustreError: 321993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.080569] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8943.094784] LustreError: 321993:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8943.103782] Lustre: Skipped 1 previous similar message [ 8947.173225] LustreError: 321992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8947.173335] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8947.182795] LustreError: 321992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8947.185318] Lustre: Skipped 1 previous similar message [ 8952.287143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8952.287747] LustreError: 321992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8952.304239] LustreError: 321992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8952.428135] Lustre: server umount lustre-MDT0001 complete [ 8957.248461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8964.183711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8981.988546] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4ed0724b152e7809 [ 8981.997970] Lustre: MGC192.168.206.108@tcp: Connection restored to 0@lo (at 0@lo) [ 8982.258063] LustreError: 323825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8982.334308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.396260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8985.789204] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8991.712843] LustreError: 323826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8991.727519] LustreError: 323826:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8992.305456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8992.616825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8996.801274] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9003.822723] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9005.779547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9007.085918] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9011.096658] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9012.819996] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9017.316427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.326066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9017.333155] Lustre: Skipped 1 previous similar message [ 9017.341549] Lustre: Skipped 2 previous similar messages [ 9022.275923] Lustre: server umount lustre-OST0000 complete [ 9027.552480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.571239] Lustre: Skipped 1 previous similar message [ 9037.797077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9037.802718] Lustre: Skipped 5 previous similar messages [ 9039.327135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9039.467372] Lustre: server umount lustre-MDT0000 complete [ 9042.183369] LustreError: 321976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955802 with bad export cookie 5679164796917676041 [ 9042.190762] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9042.196821] LustreError: 321976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9042.495927] Lustre: server umount lustre-MDT0001 complete [ 9049.606208] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9051.470518] Key type lgssc unregistered [ 9051.708575] LNet: 328283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9051.715968] LNetError: 328283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9051.732780] LNet: Removed LNI 192.168.206.108@tcp [ 9052.352881] Key type .llcrypt unregistered [ 9052.354489] Key type ._llcrypt unregistered [ 9069.130433] Key type ._llcrypt registered [ 9069.132332] Key type .llcrypt registered [ 9069.201775] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9069.677605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9069.685747] alg: No test for adler32 (adler32-zlib) [ 9070.648602] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 9070.794714] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 9072.415777] Key type lgssc registered [ 9073.151254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9077.837922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9077.850635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9079.142759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.984730] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9088.311465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9088.593244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9091.771573] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9096.700220] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9100.105948] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9103.841709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9103.849906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9103.852273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9103.856332] Lustre: Skipped 1 previous similar message [ 9108.962607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9108.966529] Lustre: Skipped 1 previous similar message [ 9113.057958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9113.064444] Lustre: Skipped 1 previous similar message [ 9116.127120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9116.247581] Lustre: server umount lustre-MDT0000 complete [ 9118.178284] LustreError: 329694:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9118.192940] LustreError: 329694:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9118.400663] LustreError: 329673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955878 with bad export cookie 16287646318502302955 [ 9118.408841] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9118.411733] LustreError: 329673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9118.634521] Lustre: server umount lustre-MDT0001 complete [ 9127.993343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9128.209860] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9128.257334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9128.403300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9131.177460] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9136.397231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.483608] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9136.508238] Lustre: 332280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9136.680375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9139.339454] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9143.772662] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9147.003838] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9153.726314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9153.846549] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9153.851810] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9153.859857] LustreError: 334164:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9153.866737] LustreError: 334164:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9153.873892] LustreError: 334164:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9153.932783] Lustre: server umount lustre-OST0000 complete [ 9153.935108] LustreError: 334164:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9162.276724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9162.404745] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9162.531414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9167.893651] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9169.039724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9176.440330] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9182.633271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9182.695714] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9182.750926] Lustre: lustre-OST0001: new disk, initializing [ 9182.753959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9182.758355] Lustre: 335704:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9182.820406] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9187.837194] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9189.466335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9189.471808] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9189.571450] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9193.322352] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9200.099992] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9200.101295] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.104920] LustreError: Skipped 1 previous similar message [ 9200.105467] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9200.113892] Lustre: Skipped 2 previous similar messages [ 9201.394919] Lustre: server umount lustre-OST0001 complete [ 9204.704447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9204.711925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9204.728183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9204.732624] Lustre: Skipped 1 previous similar message [ 9205.221645] LustreError: 334631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9205.236816] LustreError: 334631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9209.781863] Lustre: server umount lustre-OST0000 complete [ 9213.412414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9213.420325] LustreError: Skipped 1 previous similar message [ 9213.426155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.431385] Lustre: Skipped 1 previous similar message [ 9213.434425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9213.437400] Lustre: Skipped 2 previous similar messages [ 9218.563304] Lustre: server umount lustre-MDT0000 complete [ 9220.577620] LustreError: 332294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9220.585914] LustreError: 332294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9220.894457] LustreError: 332273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955980 with bad export cookie 16287646318502303508 [ 9220.897111] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9220.904728] LustreError: 332273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9221.092603] Lustre: server umount lustre-MDT0001 complete [ 9227.394647] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9229.023242] Key type lgssc unregistered [ 9229.175673] LNet: 337660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9229.181566] LNetError: 337660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9229.192820] LNet: Removed LNI 192.168.206.108@tcp [ 9229.727916] Key type .llcrypt unregistered [ 9229.729488] Key type ._llcrypt unregistered [ 9280.791771] Key type ._llcrypt registered [ 9280.794673] Key type .llcrypt registered [ 9280.881219] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 9290.589522] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9291.090401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9291.101530] alg: No test for adler32 (adler32-zlib) [ 9292.106985] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 9292.296685] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 9293.935251] Key type lgssc registered [ 9294.801101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9316.502540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9316.522947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9317.679146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9317.699973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9317.754860] Lustre: lustre-MDT0000: new disk, initializing [ 9317.817855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9317.829785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9320.512368] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9327.746299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.838983] Lustre: 342584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9327.885784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9327.890277] Lustre: Skipped 1 previous similar message [ 9327.955691] Lustre: lustre-MDT0001: new disk, initializing [ 9328.030368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9328.066925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9328.080753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9330.886634] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9337.234592] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9340.686649] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9345.137713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9345.317193] Lustre: lustre-OST0000: new disk, initializing [ 9345.320513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9345.327745] Lustre: 344550:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9345.390208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9349.201107] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9353.777068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9353.786706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9353.849238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9355.019115] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9357.852303] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9357.966278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9360.666936] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9360.816540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9363.938075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9363.942211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9363.958148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9364.452241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9364.453493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.471145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9364.475073] Lustre: Skipped 1 previous similar message [ 9369.571428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.575540] Lustre: Skipped 1 previous similar message [ 9374.689636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9374.693739] Lustre: Skipped 1 previous similar message [ 9376.223352] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9376.357891] Lustre: server umount lustre-OST0000 complete [ 9379.296877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9379.305353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.316646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9385.100634] Lustre: server umount lustre-MDT0000 complete [ 9387.399318] LustreError: 342575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956147 with bad export cookie 13611411364890831725 [ 9387.404195] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9387.410483] LustreError: 342575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9387.596500] Lustre: server umount lustre-MDT0001 complete [ 9392.629944] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:49:11 (1790956151) [ 9397.100242] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:49:16 (1790956156) [ 9402.261633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9402.613547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9405.239943] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9407.968883] LustreError: 347304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9407.985052] LustreError: 347304:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9410.593326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9410.803410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9413.701224] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9418.417713] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9421.677082] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9425.416459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9425.572584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9428.495979] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9433.337947] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9444.466963] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9445.577870] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9446.947808] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9448.419358] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9449.979422] Lustre: Failing over lustre-MDT0000 [ 9450.191866] Lustre: server umount lustre-MDT0000 complete [ 9451.424442] LustreError: 347301:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9451.498352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9451.503510] Lustre: Skipped 1 previous similar message [ 9454.401259] LustreError: 347301:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9454.424081] LustreError: 347301:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9456.616910] LustreError: 347304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9456.637029] LustreError: 347304:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9461.734326] LustreError: 350184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9461.751481] LustreError: 350184:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9466.261364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9466.367768] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9466.532185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.574726] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9467.766314] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9469.215205] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9471.974859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9471.987846] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9472.007815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9475.941976] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9477.302223] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9480.319371] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9481.824334] Lustre: Failing over lustre-MDT0000 [ 9482.083093] Lustre: server umount lustre-MDT0000 complete [ 9482.226242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9482.226703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9482.244776] Lustre: Skipped 2 previous similar messages [ 9482.250485] LustreError: 347300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9482.250495] LustreError: 347300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9498.163317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.245339] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9498.448446] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9498.502744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9500.644655] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9503.722894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9503.728459] Lustre: Skipped 2 previous similar messages [ 9503.746472] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9503.779774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9506.961413] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9507.884133] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9513.953073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9513.957372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9513.966990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9513.985657] Lustre: Skipped 2 previous similar messages [ 9513.991339] Lustre: Skipped 5 previous similar messages [ 9518.162829] Lustre: server umount lustre-MDT0000 complete [ 9519.077728] LustreError: 347305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9519.087969] LustreError: 347305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 9520.762384] LustreError: 347285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956280 with bad export cookie 13611411364890834434 [ 9520.765208] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9520.773562] LustreError: 347285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9520.997588] Lustre: server umount lustre-MDT0001 complete [ 9534.456411] Lustre: server umount lustre-OST0000 complete [ 9544.382715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.681786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.685555] Lustre: Skipped 1 previous similar message [ 9547.411356] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9552.890145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.337872] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9561.021584] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9564.866808] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9569.365645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9569.448949] LustreError: 355475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9569.462215] LustreError: 355475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9573.287482] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9574.895747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9578.044231] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9583.075880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9583.077118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.083048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9583.088537] Lustre: Skipped 1 previous similar message [ 9586.936867] Lustre: server umount lustre-OST0000 complete [ 9595.139574] Lustre: server umount lustre-MDT0000 complete [ 9597.247146] LustreError: 353638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956357 with bad export cookie 13611411364890835183 [ 9597.247559] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.261069] LustreError: 353638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9597.523982] Lustre: server umount lustre-MDT0001 complete [ 9604.565441] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9606.980768] Key type lgssc unregistered [ 9607.237377] LNet: 357183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9607.242482] LNetError: 357183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9607.265671] LNet: Removed LNI 192.168.206.108@tcp [ 9608.018926] Key type .llcrypt unregistered [ 9608.021358] Key type ._llcrypt unregistered [ 9625.599091] Key type ._llcrypt registered [ 9625.600687] Key type .llcrypt registered [ 9625.655487] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9626.192742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9626.248879] alg: No test for adler32 (adler32-zlib) [ 9627.196302] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 9627.348885] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 9628.983431] Key type lgssc registered [ 9629.689577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9634.333598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9634.347687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9635.662603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9638.044450] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9640.929805] LustreError: 358694:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9640.947802] LustreError: 358694:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9644.457150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.765499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9647.361451] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9652.279405] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9656.843532] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9661.582213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9661.868541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.598272] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9671.543317] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9680.304103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9690.592688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9690.595979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9690.600260] Lustre: Skipped 2 previous similar messages [ 9690.603118] Lustre: Skipped 1 previous similar message [ 9695.082265] Lustre: server umount lustre-MDT0000 complete [ 9695.713344] LustreError: 358690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.721909] LustreError: 358690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9697.191618] LustreError: 358676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956457 with bad export cookie 15059959853066501768 [ 9697.195533] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9697.201967] LustreError: 358676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9697.401528] Lustre: server umount lustre-MDT0001 complete [ 9709.885847] Lustre: server umount lustre-OST0000 complete [ 9717.994507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.260542] LustreError: 362423:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.315750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9720.950391] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9723.360747] LustreError: 362424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.620084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9726.859225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9729.381575] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9733.614776] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9736.844279] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9740.845668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9741.054599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.153278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9745.026483] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9751.076737] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9761.763832] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9761.769262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9761.772747] Lustre: Skipped 1 previous similar message [ 9761.783299] Lustre: Skipped 2 previous similar messages [ 9766.861570] Lustre: server umount lustre-MDT0000 complete [ 9766.889581] LustreError: 362424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9769.036993] LustreError: 362406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956529 with bad export cookie 15059959853066502881 [ 9769.042173] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9769.044747] LustreError: 362406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9769.276819] Lustre: server umount lustre-MDT0001 complete [ 9781.935981] Lustre: server umount lustre-OST0000 complete [ 9793.180746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9793.418654] LustreError: 366328:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9793.428805] LustreError: 366328:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9793.471481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9795.958579] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9801.573957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9804.796105] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9810.419071] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9814.284298] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9819.129054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9823.728478] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9828.698301] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9838.898204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9844.200572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9844.201915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9844.204292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9844.217926] Lustre: Skipped 2 previous similar messages [ 9844.924398] Lustre: server umount lustre-OST0000 complete [ 9849.317626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9849.326654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9849.329233] Lustre: Skipped 1 previous similar message [ 9854.434819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9854.440302] Lustre: Skipped 1 previous similar message [ 9862.111678] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9862.197215] Lustre: server umount lustre-MDT0000 complete [ 9864.208710] LustreError: 367065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956624 with bad export cookie 15059959853066503903 [ 9864.216348] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9864.219074] LustreError: 367065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9864.385580] Lustre: server umount lustre-MDT0001 complete [ 9870.386512] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9872.161886] Key type lgssc unregistered [ 9872.395669] LNet: 369880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9872.408201] LNetError: 369880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9872.423633] LNet: Removed LNI 192.168.206.108@tcp [ 9872.900073] Key type .llcrypt unregistered [ 9872.901684] Key type ._llcrypt unregistered [ 9888.744240] Key type ._llcrypt registered [ 9888.747322] Key type .llcrypt registered [ 9888.820217] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9889.360495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9889.384015] alg: No test for adler32 (adler32-zlib) [ 9890.331317] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 9890.486668] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 9892.127276] Key type lgssc registered [ 9893.029392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9898.415502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9898.429487] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.436958] LustreError: 371316:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9898.450654] LustreError: 371316:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9898.455933] LustreError: 371316:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9898.461187] LustreError: 371316:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9898.468418] LustreError: 371316:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9903.339395] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9903.345594] LustreError: 371713:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9903.361537] LustreError: 371713:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9903.368964] LustreError: 371713:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9903.374899] LustreError: 371713:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9903.380642] LustreError: 371713:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9915.991805] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 9917.851106] Key type lgssc unregistered [ 9918.055684] LNet: 372714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9918.059668] LNetError: 372714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9918.069042] LNet: Removed LNI 192.168.206.108@tcp [ 9918.622525] Key type .llcrypt unregistered [ 9918.627134] Key type ._llcrypt unregistered [ 9930.459465] Key type ._llcrypt registered [ 9930.464560] Key type .llcrypt registered [ 9930.541931] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 9940.544403] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 9940.975401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9941.026659] alg: No test for adler32 (adler32-zlib) [ 9941.958485] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 9942.092162] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 9943.711223] Key type lgssc registered [ 9944.286814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9963.864050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9963.879192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9965.015594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9965.042125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9965.094259] Lustre: lustre-MDT0000: new disk, initializing [ 9965.148092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.166167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9967.396670] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9973.469253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.527075] Lustre: 375862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9973.559775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9973.562668] Lustre: Skipped 1 previous similar message [ 9973.617643] Lustre: lustre-MDT0001: new disk, initializing [ 9973.673798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9973.700416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9973.706679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9975.993342] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9981.549742] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9984.547380] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9988.201396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9988.351580] Lustre: lustre-OST0000: new disk, initializing [ 9988.354196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9988.358369] Lustre: 377832:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9988.409315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9991.688936] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 9996.806777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9996.811159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9996.864379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9996.880371] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9998.804540] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9998.921742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10000.792774] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10000.886319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10001.891153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10001.891616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10001.892085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10001.895318] Lustre: Skipped 1 previous similar message [10001.899489] LustreError: Skipped 1 previous similar message [10003.921654] Lustre: server umount lustre-OST0000 complete [10007.009033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10007.010573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10007.017856] Lustre: Skipped 1 previous similar message [10007.023804] Lustre: Skipped 2 previous similar messages [10012.130993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10012.136446] Lustre: Skipped 1 previous similar message [10012.323104] Lustre: server umount lustre-MDT0000 complete [10014.391465] LustreError: 375853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956774 with bad export cookie 16200587110439373680 [10014.395213] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10014.396861] LustreError: 375853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10014.581778] Lustre: server umount lustre-MDT0001 complete [10018.629927] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10019.440958] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:59:38 (1790956778) [10022.781225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.075384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10025.040521] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10028.512450] LustreError: 380287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10028.522139] LustreError: 380287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10029.251103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10029.499116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10031.417439] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10034.679710] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10037.145102] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10040.316330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10040.479430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10043.124395] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10046.885487] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10050.212247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10050.293596] Lustre: lustre-OST0001: new disk, initializing [10050.297647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10050.301614] Lustre: 383119:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10052.964431] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10057.696655] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10058.265716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10058.270959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10058.330647] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10063.333330] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10063.333534] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10063.339219] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10063.341974] LustreError: Skipped 1 previous similar message [10066.058202] Lustre: server umount lustre-OST0001 complete [10068.452318] LustreError: 383927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10068.464108] LustreError: 383927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10073.570883] LustreError: 383837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10073.577879] LustreError: 383837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10078.695069] LustreError: 382107:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10078.701798] LustreError: 382107:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10079.714382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10079.720450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.727081] Lustre: Skipped 1 previous similar message [10079.730982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10079.734384] Lustre: Skipped 2 previous similar messages [10083.810732] LustreError: 382106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10083.820203] LustreError: 382106:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10084.969264] Lustre: server umount lustre-OST0000 complete [10088.929807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10088.930988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10088.935067] Lustre: Skipped 2 previous similar messages [10088.939923] Lustre: Skipped 4 previous similar messages [10092.829843] Lustre: server umount lustre-MDT0000 complete [10094.048232] LustreError: 380287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10094.056844] LustreError: 380287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10094.362414] LustreError: 380267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956854 with bad export cookie 16200587110439374569 [10094.363329] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10094.367652] LustreError: 380267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10100.556165] Lustre: server umount lustre-MDT0001 complete [10105.257919] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10106.505181] Key type lgssc unregistered [10106.666486] LNet: 385115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10106.669898] LNetError: 385115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10106.679567] LNet: Removed LNI 192.168.206.108@tcp [10107.077193] Key type .llcrypt unregistered [10107.079405] Key type ._llcrypt unregistered [10130.069140] Key type ._llcrypt registered [10130.070808] Key type .llcrypt registered [10130.121241] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10130.569612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10130.618487] alg: No test for adler32 (adler32-zlib) [10131.496270] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10131.612817] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10133.223234] Key type lgssc registered [10133.825543] Lustre: Echo OBD driver; http://www.lustre.org/ [10137.394401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10137.404841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10138.524044] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10138.536067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10138.607415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10140.414392] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10144.157339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10144.215207] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10144.232728] Lustre: 387904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10144.378874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10146.200068] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10149.285550] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10151.507087] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10154.464164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10154.467600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10154.474819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10157.024124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10157.024836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.556433] Lustre: server umount lustre-MDT0000 complete [10160.209662] LustreError: 388645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956920 with bad export cookie 17769185734568482323 [10160.211530] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10160.214360] LustreError: 388645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10160.346673] Lustre: server umount lustre-MDT0001 complete [10164.360916] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:02:03 (1790956923) [10167.045926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10167.281937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10169.144930] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10172.386240] LustreError: 390416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.396062] LustreError: 390416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10173.175854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.343377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10175.195635] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10178.406817] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10180.912126] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10183.750821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10183.845379] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10183.919028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10186.263678] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10189.271954] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10198.841991] Lustre: Permanently deactivating lustre-OST0000 [10198.849707] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10203.637744] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10205.120969] LustreError: 393403:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10205.125113] LustreError: 393403:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10205.130700] LustreError: 393403:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10206.075988] LustreError: 393451:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10206.079904] LustreError: 393451:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10212.957289] Lustre: server umount lustre-OST0000 complete [10214.093589] LustreError: 393700:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10214.098402] LustreError: 393700:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10219.489525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10219.489529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10219.490872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10219.501487] Lustre: Skipped 2 previous similar messages [10224.608470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10224.611377] Lustre: Skipped 1 previous similar message [10229.215257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10229.293506] Lustre: server umount lustre-MDT0000 complete [10229.729295] LustreError: 390417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10229.737043] LustreError: 390417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10230.858975] LustreError: 390397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956990 with bad export cookie 17769185734568482897 [10230.861651] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10230.865562] LustreError: 390397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10240.030085] Lustre: server umount lustre-MDT0001 complete [10243.707423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.488077] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10246.643140] LustreError: 394858:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10249.305418] Lustre: 395196:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.108@tcp [10250.154441] Lustre: server umount MGS complete [10255.039669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10255.285300] LustreError: 395919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.298133] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10255.301606] Lustre: Skipped 1 previous similar message [10255.330398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10257.145316] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10260.449099] LustreError: 395920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.032816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10261.169509] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10263.106656] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10266.747398] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10269.442668] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10270.325184] Lustre: Permanently reactivating lustre-OST0000 [10270.331987] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10270.335507] Lustre: Skipped 2 previous similar messages [10294.239100] LustreError: 397650:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10296.663840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10296.793928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10296.796788] Lustre: Skipped 1 previous similar message [10298.738336] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10301.326904] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10314.210125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10314.211294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10314.212624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10314.227808] Lustre: Skipped 2 previous similar messages [10315.668424] Lustre: server umount lustre-OST0000 complete [10317.792633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10319.201904] Lustre: server umount lustre-MDT0000 complete [10319.328076] LustreError: 395919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.328860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10319.338478] Lustre: Skipped 1 previous similar message [10320.728828] LustreError: 396654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957080 with bad export cookie 17769185734568483933 [10320.733564] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10320.733897] LustreError: 396654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10320.869289] Lustre: server umount lustre-MDT0001 complete [10325.268671] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10326.401397] Key type lgssc unregistered [10326.544557] LNet: 400442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10326.547877] LNetError: 400442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10326.557344] LNet: Removed LNI 192.168.206.108@tcp [10326.925635] Key type .llcrypt unregistered [10326.926994] Key type ._llcrypt unregistered [10334.025829] Key type ._llcrypt registered [10334.027480] Key type .llcrypt registered [10334.072049] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [10341.054688] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10341.400912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10341.455648] alg: No test for adler32 (adler32-zlib) [10342.331356] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10342.431623] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10344.015159] Key type lgssc registered [10344.441772] Lustre: Echo OBD driver; http://www.lustre.org/ [10357.681797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10357.694203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.776390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10358.788552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10358.830568] Lustre: lustre-MDT0000: new disk, initializing [10358.858540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10358.864476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10360.314530] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10364.629822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10364.671626] Lustre: 403586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10364.695613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10364.698248] Lustre: Skipped 1 previous similar message [10364.734346] Lustre: lustre-MDT0001: new disk, initializing [10364.769052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10364.782877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10364.788122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10366.248122] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10370.107967] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10372.059281] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10374.388841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10374.488471] Lustre: lustre-OST0000: new disk, initializing [10374.490587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10374.494338] Lustre: 405553:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10374.528252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10376.735945] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10380.280604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10380.284912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10380.320692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10380.806694] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10382.367233] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10382.439835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10383.870991] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10383.955487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10385.376984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10385.377655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.380025] LustreError: Skipped 1 previous similar message [10385.386021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10385.389793] Lustre: Skipped 1 previous similar message [10386.717100] Lustre: server umount lustre-OST0000 complete [10390.497862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10390.498301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10390.502182] Lustre: Skipped 1 previous similar message [10390.505675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10390.509928] Lustre: Skipped 1 previous similar message [10394.205485] Lustre: server umount lustre-MDT0000 complete [10395.619493] LustreError: 403577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957155 with bad export cookie 2195568081795898539 [10395.623705] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10395.624145] LustreError: 403577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10395.762799] Lustre: server umount lustre-MDT0001 complete [10399.285628] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10400.074388] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:05:59 (1790957159) [10402.776884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10403.021446] LustreError: 408007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10403.031148] LustreError: 408007:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10403.063983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10404.726465] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10408.024740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10408.163758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10409.705813] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10412.756632] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10415.048243] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10417.720769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10417.873427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10420.124687] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10422.884117] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10423.700776] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10429.421234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10433.508039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10433.509635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10433.509954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10433.513099] Lustre: Skipped 1 previous similar message [10437.083807] Lustre: server umount lustre-OST0000 complete [10438.624356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.625122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.628866] Lustre: Skipped 1 previous similar message [10438.635304] Lustre: Skipped 2 previous similar messages [10443.744153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10443.747315] Lustre: Skipped 1 previous similar message [10444.696048] Lustre: server umount lustre-MDT0000 complete [10446.164458] LustreError: 407987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957206 with bad export cookie 2195568081795899407 [10446.168123] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10446.169293] LustreError: 407987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10446.300562] Lustre: server umount lustre-MDT0001 complete [10450.660635] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10451.832622] Key type lgssc unregistered [10451.970558] LNet: 411729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10451.974487] LNetError: 411729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10451.986530] LNet: Removed LNI 192.168.206.108@tcp [10452.331267] Key type .llcrypt unregistered [10452.332843] Key type ._llcrypt unregistered [10461.444367] Key type ._llcrypt registered [10461.446118] Key type .llcrypt registered [10461.494645] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10461.635225] Key type .llcrypt unregistered [10461.637367] Key type ._llcrypt unregistered [10469.766146] Key type ._llcrypt registered [10469.767996] Key type .llcrypt registered [10469.809377] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10470.147704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10470.208172] alg: No test for adler32 (adler32-zlib) [10471.078582] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10471.179604] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10472.767191] Key type lgssc registered [10473.203127] Lustre: Echo OBD driver; http://www.lustre.org/ [10476.259739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10476.269650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10477.468885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10479.102913] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10482.334390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10482.460132] LustreError: 413929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10482.464562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10482.467436] LustreError: 413929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10484.606932] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10487.315620] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10487.584521] LustreError: 413933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10487.590114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10487.590897] LustreError: 413933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10489.900531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.073282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10491.666150] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10500.578135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10500.578584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10500.581082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10500.586080] Lustre: Skipped 1 previous similar message [10500.590015] Lustre: Skipped 1 previous similar message [10505.494725] Lustre: server umount lustre-OST0000 complete [10509.793775] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10509.796822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10509.802740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10510.816176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10510.816639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.059190] Lustre: server umount lustre-MDT0000 complete [10514.454412] LustreError: 415675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957274 with bad export cookie 2194372720987203888 [10514.457471] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10514.459412] LustreError: 415675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10514.578491] Lustre: server umount lustre-MDT0001 complete [10518.822709] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10519.860978] Key type lgssc unregistered [10519.991392] LNet: 417077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10519.994696] LNetError: 417077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10520.007321] LNet: Removed LNI 192.168.206.108@tcp [10520.344521] Key type .llcrypt unregistered [10520.346292] Key type ._llcrypt unregistered [10530.652451] Key type ._llcrypt registered [10530.653725] Key type .llcrypt registered [10530.697522] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10531.065824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10531.099087] alg: No test for adler32 (adler32-zlib) [10531.964726] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10532.065636] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10533.655197] Key type lgssc registered [10534.099379] Lustre: Echo OBD driver; http://www.lustre.org/ [10537.004924] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10537.016156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10552.479318] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10557.599265] LustreError: 418458:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.108@tcp: failed processing log, type 4: rc = -110 [10584.223161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10586.336827] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10589.061602] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10591.586972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10591.793519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10591.820654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10593.313295] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10596.385073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10596.521772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10598.042234] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10600.716708] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10602.994283] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10607.072993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10607.073291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10607.075955] LustreError: Skipped 1 previous similar message [10607.076816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10607.081240] Lustre: Skipped 1 previous similar message [10607.087321] Lustre: Skipped 1 previous similar message [10610.145264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10610.150389] Lustre: Skipped 1 previous similar message [10611.796136] Lustre: server umount lustre-OST0000 complete [10615.265759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.266501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10615.270127] Lustre: Skipped 1 previous similar message [10615.275348] Lustre: Skipped 1 previous similar message [10619.229291] Lustre: server umount lustre-MDT0000 complete [10620.384766] LustreError: 419498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.392450] LustreError: 419498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10620.646803] LustreError: 418466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957380 with bad export cookie 15101090700572924318 [10620.649630] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10620.651765] LustreError: 418466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10620.771609] Lustre: server umount lustre-MDT0001 complete [10624.963734] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10626.026543] Key type lgssc unregistered [10626.155437] LNet: 422024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10626.158349] LNetError: 422024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10626.167277] LNet: Removed LNI 192.168.206.108@tcp [10626.510236] Key type .llcrypt unregistered [10626.511647] Key type ._llcrypt unregistered [10636.921391] Key type ._llcrypt registered [10636.922576] Key type .llcrypt registered [10636.967915] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10637.299752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10637.329054] alg: No test for adler32 (adler32-zlib) [10638.183355] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10638.286920] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10639.871154] Key type lgssc registered [10640.322031] Lustre: Echo OBD driver; http://www.lustre.org/ [10643.127346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10643.135607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10644.288769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10645.654248] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10648.543693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.667341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10650.007449] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10652.486584] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10654.490175] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10656.806000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10656.918358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10658.918851] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10661.539302] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10670.247643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10674.144173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10674.148305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10674.155827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10675.680344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10675.681818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10675.690039] Lustre: Skipped 1 previous similar message [10677.129198] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [10677.132692] Lustre: Skipped 2 previous similar messages [10679.205451] Lustre: server umount lustre-MDT0000 complete [10680.027747] LustreError: 423413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.8@tcp arrived at 1790957440 with bad export cookie 17724654760678739343 [10680.032842] LustreError: 423413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10680.709786] LustreError: 423412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957440 with bad export cookie 17724654760678738692 [10680.713730] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10680.714564] LustreError: 423412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10680.799906] LustreError: 424863:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.800478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10680.807675] LustreError: 424863:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10680.812835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10685.919797] LustreError: 423431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10685.920221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10685.925872] LustreError: 423431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10686.805070] Lustre: server umount lustre-OST0000 complete [10689.332507] Lustre: server umount lustre-MDT0001 complete [10693.266238] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10694.369521] Key type lgssc unregistered [10694.515555] LNet: 427132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10694.518751] LNetError: 427132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10694.528408] LNet: Removed LNI 192.168.206.108@tcp [10694.874624] Key type .llcrypt unregistered [10694.875619] Key type ._llcrypt unregistered [10704.896073] Key type ._llcrypt registered [10704.897512] Key type .llcrypt registered [10704.938640] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10705.308494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10705.343925] alg: No test for adler32 (adler32-zlib) [10706.195717] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10706.290850] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10707.871167] Key type lgssc registered [10708.267306] Lustre: Echo OBD driver; http://www.lustre.org/ [10710.954050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10710.962305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.108934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10713.467464] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10716.373418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10716.488877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10717.878095] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10720.387424] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10722.398301] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10725.014473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10725.147478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10727.221773] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10729.906972] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10737.445191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10742.239901] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10742.243967] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.250233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10742.753603] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.759708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10745.736639] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.8@tcp (stopping) [10747.664123] Lustre: server umount lustre-MDT0001 complete [10747.873803] LustreError: 428538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10747.879816] LustreError: 428538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10752.992405] LustreError: 428537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10752.993042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10752.993240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10752.994148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10755.412328] Lustre: server umount lustre-OST0000 complete [10756.810152] Lustre: server umount lustre-MDT0000 complete [10761.444207] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10762.546336] Key type lgssc unregistered [10762.672441] LNet: 432242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10762.674845] LNetError: 432242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10762.682320] LNet: Removed LNI 192.168.206.108@tcp [10763.003368] Key type .llcrypt unregistered [10763.004443] Key type ._llcrypt unregistered [10771.420546] Key type ._llcrypt registered [10771.421891] Key type .llcrypt registered [10771.461134] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [10778.116417] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10778.697631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10778.717571] alg: No test for adler32 (adler32-zlib) [10779.587827] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10779.676251] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10781.263165] Key type lgssc registered [10781.680141] Lustre: Echo OBD driver; http://www.lustre.org/ [10794.847228] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10799.997513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10800.003940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.083824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10801.094167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10801.129678] Lustre: lustre-MDT0000: new disk, initializing [10801.152321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10801.158847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10802.542252] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10808.464215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10808.573282] Lustre: lustre-OST0000: new disk, initializing [10808.575825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10808.578179] Lustre: Skipped 1 previous similar message [10808.580575] Lustre: 437263:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10808.609638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10810.676538] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10814.563610] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10816.512533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10816.515783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10816.523794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10825.224065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10825.270245] Lustre: 436243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10825.303679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10825.343922] Lustre: lustre-MDT0001: new disk, initializing [10825.388105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10825.402327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10825.405877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10826.731289] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10830.303708] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10830.375980] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10831.851559] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10836.000878] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 4 sec [10837.894689] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.8@tcp (stopping) [10840.033415] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.033655] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10840.034307] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10840.036912] Lustre: Skipped 1 previous similar message [10843.006569] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.8@tcp (stopping) [10843.009989] Lustre: Skipped 1 previous similar message [10843.664585] Lustre: server umount lustre-MDT0001 complete [10850.272933] LustreError: 439585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.278287] LustreError: 439585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10855.394805] LustreError: 437271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10855.403767] LustreError: 437271:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10857.439884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10857.443457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10857.449734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10860.512895] LustreError: 436255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10865.632498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10865.633723] LustreError: 439594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10865.634924] Lustre: Skipped 1 previous similar message [10870.239152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10870.305243] Lustre: server umount lustre-OST0000 complete [10870.752773] LustreError: 436255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10877.944744] Lustre: server umount lustre-MDT0000 complete [10882.664809] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10883.704226] Key type lgssc unregistered [10883.825517] LNet: 441035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10883.828480] LNetError: 441035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10883.838274] LNet: Removed LNI 192.168.206.108@tcp [10884.157935] Key type .llcrypt unregistered [10884.159327] Key type ._llcrypt unregistered [10898.063060] Key type ._llcrypt registered [10898.064246] Key type .llcrypt registered [10898.111965] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [10898.218367] Key type .llcrypt unregistered [10898.219611] Key type ._llcrypt unregistered [10905.589413] Key type ._llcrypt registered [10905.590742] Key type .llcrypt registered [10905.627528] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [10905.961336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10905.998548] alg: No test for adler32 (adler32-zlib) [10906.872880] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10906.965056] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [10908.551144] Key type lgssc registered [10909.005550] Lustre: Echo OBD driver; http://www.lustre.org/ [10918.536939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10918.546244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10919.649827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10919.661220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10919.699212] Lustre: lustre-MDT0000: new disk, initializing [10919.728370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10919.737659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10921.306428] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10926.087685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10926.137278] Lustre: 444396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10926.164389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10926.166777] Lustre: Skipped 1 previous similar message [10926.205841] Lustre: lustre-MDT0001: new disk, initializing [10926.247744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10926.263578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10926.267283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10927.759117] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10931.677426] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10933.864286] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10936.347363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10936.458374] Lustre: lustre-OST0000: new disk, initializing [10936.460753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10936.464344] Lustre: 446366:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10936.501651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10938.598394] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10942.801807] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10945.528267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10945.531867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10945.564059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10952.160038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10952.162381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10952.166455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10955.745569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10955.746693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10955.749589] Lustre: Skipped 1 previous similar message [10955.753280] Lustre: Skipped 2 previous similar messages [10956.961478] Lustre: server umount lustre-MDT0000 complete [10958.300253] LustreError: 444390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957718 with bad export cookie 989003742438221968 [10958.303972] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10958.304548] LustreError: 444390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10958.423270] Lustre: server umount lustre-MDT0001 complete [10969.073205] Lustre: server umount lustre-OST0000 complete [10972.843835] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:15:32 (1790957732) [10975.438737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.612531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10977.106517] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10980.335676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10980.463357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10981.857096] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [10984.461604] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10986.353370] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11001.633938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11001.701097] Lustre: Found index 0 for lustre-OST0000, updating log [11001.762863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11003.564907] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11005.852204] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11021.222239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11026.400513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11026.402326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11026.403263] LustreError: Skipped 1 previous similar message [11026.407573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11026.410201] Lustre: Skipped 1 previous similar message [11029.004045] Lustre: server umount lustre-OST0000 complete [11041.760057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11041.763038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11041.766433] Lustre: Skipped 1 previous similar message [11041.768446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11041.770914] Lustre: Skipped 2 previous similar messages [11047.697780] Lustre: server umount lustre-MDT0000 complete [11048.869967] LustreError: 450241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957808 with bad export cookie 989003742438223347 [11048.874136] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11048.983430] Lustre: server umount lustre-MDT0001 complete [11051.839510] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:16:51 (1790957811) [11066.719269] LNet: 1 local NIs in recovery (showing 1): 192.168.206.108@tcp [11630.834160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11630.972839] LustreError: 454812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11630.979188] LustreError: 454812:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11630.996547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11632.248517] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11634.992940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11635.103641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11636.395644] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11638.769588] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11640.492109] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11655.585938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11655.676955] Lustre: Found index 0 for lustre-OST0000, updating log [11655.786940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11657.768100] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11658.854197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11660.292291] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11661.984735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11661.988027] Lustre: Skipped 1 previous similar message [11661.989662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11661.991446] Lustre: Skipped 2 previous similar messages [11663.009229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11667.080538] Lustre: server umount lustre-OST0000 complete [11669.216863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11669.218515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11669.219972] Lustre: Skipped 1 previous similar message [11671.264114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11671.266119] Lustre: Skipped 4 previous similar messages [11674.328922] Lustre: server umount lustre-MDT0000 complete [11674.337661] LustreError: 455539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11674.345590] LustreError: 455539:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11675.360028] LustreError: 454812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.364341] LustreError: 454812:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11675.592132] LustreError: 456714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958435 with bad export cookie 989003742438224418 [11675.595404] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11675.596551] LustreError: 456714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11675.711653] Lustre: server umount lustre-MDT0001 complete [11682.143322] LNet: 1 local NIs in recovery (showing 1): 192.168.206.108@tcp [11693.032052] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [11696.220246] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11707.182404] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:27:46 (1790958466) [11712.733626] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11713.737513] Key type lgssc unregistered [11719.007483] LNet: 461464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11719.010716] LNetError: 461464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11785.573451] LNet: Removed LNI 192.168.206.108@tcp [11785.934751] Key type .llcrypt unregistered [11785.936336] Key type ._llcrypt unregistered [11792.560388] Key type ._llcrypt registered [11792.561303] Key type .llcrypt registered [11792.596468] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11793.024890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11793.070129] alg: No test for adler32 (adler32-zlib) [11793.926944] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [11794.017087] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [11795.599113] Key type lgssc registered [11795.951980] Lustre: Echo OBD driver; http://www.lustre.org/ [11797.285755] LNet: Added LNI 192.168.206.108@tcp100 [8/256/0/180] [11797.560982] LNet: Added LNI 192.168.206.108@tcp101 [8/256/0/180] [11797.830886] LNet: Added LNI 192.168.206.108@tcp102 [8/256/0/180] [11798.110470] LNet: Added LNI 192.168.206.108@tcp103 [8/256/0/180] [11798.383186] LNet: Added LNI 192.168.206.108@tcp104 [8/256/0/180] [11798.656642] LNet: Added LNI 192.168.206.108@tcp105 [8/256/0/180] [11798.933303] LNet: Added LNI 192.168.206.108@tcp106 [8/256/0/180] [11799.214221] LNet: Added LNI 192.168.206.108@tcp107 [8/256/0/180] [11799.480593] LNet: Added LNI 192.168.206.108@tcp108 [8/256/0/180] [11799.757903] LNet: Added LNI 192.168.206.108@tcp109 [8/256/0/180] [11800.036092] LNet: Added LNI 192.168.206.108@tcp110 [8/256/0/180] [11800.299099] LNet: Added LNI 192.168.206.108@tcp111 [8/256/0/180] [11800.570043] LNet: Added LNI 192.168.206.108@tcp112 [8/256/0/180] [11800.861499] LNet: Added LNI 192.168.206.108@tcp113 [8/256/0/180] [11801.151968] LNet: Added LNI 192.168.206.108@tcp114 [8/256/0/180] [11801.443196] LNet: Added LNI 192.168.206.108@tcp115 [8/256/0/180] [11801.730203] LNet: Added LNI 192.168.206.108@tcp116 [8/256/0/180] [11802.005206] LNet: Added LNI 192.168.206.108@tcp117 [8/256/0/180] [11802.274026] LNet: Added LNI 192.168.206.108@tcp118 [8/256/0/180] [11802.557204] LNet: Added LNI 192.168.206.108@tcp119 [8/256/0/180] [11802.830768] LNet: Added LNI 192.168.206.108@tcp120 [8/256/0/180] [11803.104505] LNet: Added LNI 192.168.206.108@tcp121 [8/256/0/180] [11803.374835] LNet: Added LNI 192.168.206.108@tcp122 [8/256/0/180] [11803.673066] LNet: Added LNI 192.168.206.108@tcp123 [8/256/0/180] [11803.963674] LNet: Added LNI 192.168.206.108@tcp124 [8/256/0/180] [11804.237320] LNet: Added LNI 192.168.206.108@tcp125 [8/256/0/180] [11804.503366] LNet: Added LNI 192.168.206.108@tcp126 [8/256/0/180] [11804.772537] LNet: Added LNI 192.168.206.108@tcp127 [8/256/0/180] [11805.054568] LNet: Added LNI 192.168.206.108@tcp128 [8/256/0/180] [11805.325115] LNet: Added LNI 192.168.206.108@tcp129 [8/256/0/180] [11805.581772] LNet: Added LNI 192.168.206.108@tcp130 [8/256/0/180] [11805.864318] LNet: Added LNI 192.168.206.108@tcp131 [8/256/0/180] [11806.151934] LNet: Added LNI 192.168.206.108@tcp132 [8/256/0/180] [11806.447911] LNet: Added LNI 192.168.206.108@tcp133 [8/256/0/180] [11806.730507] LNet: Added LNI 192.168.206.108@tcp134 [8/256/0/180] [11806.996331] LNet: Added LNI 192.168.206.108@tcp135 [8/256/0/180] [11809.031056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11809.038721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11810.074986] Lustre: 464477:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.108@tcp: can't add NID 192.168.206.108@tcp130: rc = -75 [11810.101419] LNetError: 464504:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [11810.104949] LNetError: Protocol error connecting to 192.168.206.108@tcp118 at host 192.168.206.108:988: is it running a compatible version of Lustre? [11810.108025] LNetError: Skipped 3 previous similar messages [11810.151077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11810.161599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11810.199316] Lustre: lustre-MDT0000: new disk, initializing [11810.221681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11810.227560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11810.399097] LNet: 9 local NIs in recovery (showing 9): 192.168.206.108@tcp122, 192.168.206.108@tcp123, 192.168.206.108@tcp124, 192.168.206.108@tcp126, 192.168.206.108@tcp125, 192.168.206.108@tcp118, 192.168.206.108@tcp127, 192.168.206.108@tcp128, 192.168.206.108@tcp129 [11811.169869] LNetError: 461932:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [11811.172848] LNetError: 461932:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [11811.175606] LNetError: Protocol error connecting to 192.168.206.108@tcp121 at host 192.168.206.108:988: is it running a compatible version of Lustre? [11811.179675] LNetError: Skipped 6 previous similar messages [11811.617093] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11813.475589] LNetError: 461931:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [11813.478468] LNetError: 461931:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [11813.486606] LNetError: Protocol error connecting to 192.168.206.108@tcp126 at host 192.168.206.108:988: is it running a compatible version of Lustre? [11813.489352] LNetError: Skipped 9 previous similar messages [11815.676163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11815.719963] Lustre: 464494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11815.745193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11815.746743] Lustre: Skipped 1 previous similar message [11815.781748] Lustre: lustre-MDT0001: new disk, initializing [11815.791137] LNetError: 464499:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [11815.795452] Lustre: 465295:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.108@tcp: can't add NID 192.168.206.108@tcp130: rc = -75 [11815.795722] LNetError: 464499:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [11815.800233] Lustre: 465295:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11815.805720] LNetError: Protocol error connecting to 192.168.206.108@tcp130 at host 192.168.206.108:988: is it running a compatible version of Lustre? [11815.810287] LNetError: Skipped 5 previous similar messages [11815.843621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11815.856759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11815.861295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11817.112752] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11819.569778] Lustre: 465775:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.108@tcp: can't add NID 192.168.206.108@tcp130: rc = -75 [11819.572288] Lustre: 465775:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11820.725052] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11822.686697] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11822.688368] LNetError: 461929:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [11822.694817] LNetError: 461929:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 15 previous similar messages [11822.697685] LNetError: Protocol error connecting to 192.168.206.108@tcp134 at host 192.168.206.108:988: is it running a compatible version of Lustre? [11822.701584] LNetError: Skipped 15 previous similar messages [11825.049987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11825.167229] Lustre: lustre-OST0000: new disk, initializing [11825.169722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11825.173568] Lustre: 466467:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11825.215708] Lustre: 466472:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.108@tcp: can't add NID 192.168.206.108@tcp130: rc = -75 [11825.220736] Lustre: 466472:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11825.241186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11826.754983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11826.759580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11826.802816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11827.301078] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11830.880420] LNetError: 461932:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [11830.883603] LNetError: 461932:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [11830.885733] LNetError: Protocol error connecting to 192.168.206.108@tcp134 at host 192.168.206.108:988: is it running a compatible version of Lustre? [11830.888677] LNetError: Skipped 1 previous similar message [11831.127110] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11836.895752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11836.898307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.902730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11837.409569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11837.410122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11837.413103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11839.629352] Lustre: server umount lustre-OST0000 complete [11841.503735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11841.506362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11841.509645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11841.511857] Lustre: Skipped 1 previous similar message [11847.264892] LNetError: 461929:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [11847.269045] LNetError: 461929:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [11847.272532] LNetError: Protocol error connecting to 192.168.206.108@tcp124 at host 192.168.206.108:988: is it running a compatible version of Lustre? [11847.276071] LNetError: Skipped 7 previous similar messages [11847.648338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11847.650029] Lustre: Skipped 3 previous similar messages [11852.767818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11852.769592] Lustre: Skipped 1 previous similar message [11855.327144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11855.385372] Lustre: server umount lustre-MDT0000 complete [11856.623036] LustreError: 465294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958616 with bad export cookie 14747895596032619553 [11856.626330] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11856.626970] LustreError: 465294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11856.745521] Lustre: server umount lustre-MDT0001 complete [11860.313714] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11861.311808] Key type lgssc unregistered [11861.474725] LNet: 468379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11861.477661] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11861.480093] LNet: Removed LNI 192.168.206.108@tcp [11862.495131] LNet: Removed LNI 192.168.206.108@tcp100 [11862.496709] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11862.499331] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11862.502267] LNet: Removed LNI 192.168.206.108@tcp101 [11863.519154] LNet: Removed LNI 192.168.206.108@tcp102 [11863.520936] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11863.524053] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11863.527356] LNet: Removed LNI 192.168.206.108@tcp103 [11864.543170] LNet: Removed LNI 192.168.206.108@tcp104 [11865.567129] LNet: Removed LNI 192.168.206.108@tcp105 [11865.568550] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11865.570480] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11865.573131] LNet: Removed LNI 192.168.206.108@tcp106 [11866.591135] LNet: Removed LNI 192.168.206.108@tcp107 [11867.615146] LNet: Removed LNI 192.168.206.108@tcp108 [11868.639131] LNet: Removed LNI 192.168.206.108@tcp109 [11869.663150] LNet: Removed LNI 192.168.206.108@tcp110 [11869.664576] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11869.667428] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11869.670350] LNet: Removed LNI 192.168.206.108@tcp111 [11870.687271] LNet: Removed LNI 192.168.206.108@tcp112 [11871.711286] LNet: Removed LNI 192.168.206.108@tcp113 [11872.735154] LNet: Removed LNI 192.168.206.108@tcp114 [11873.759288] LNet: Removed LNI 192.168.206.108@tcp115 [11874.783149] LNet: Removed LNI 192.168.206.108@tcp116 [11875.807148] LNet: Removed LNI 192.168.206.108@tcp117 [11876.831200] LNet: Removed LNI 192.168.206.108@tcp118 [11877.855149] LNet: Removed LNI 192.168.206.108@tcp119 [11877.857268] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11877.860513] LNetError: 468379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11877.864340] LNet: Removed LNI 192.168.206.108@tcp120 [11878.880290] LNet: Removed LNI 192.168.206.108@tcp121 [11879.903134] LNet: Removed LNI 192.168.206.108@tcp122 [11880.927249] LNet: Removed LNI 192.168.206.108@tcp123 [11881.951131] LNet: Removed LNI 192.168.206.108@tcp124 [11882.975140] LNet: Removed LNI 192.168.206.108@tcp125 [11882.976474] LNet: Removed LNI 192.168.206.108@tcp126 [11883.999131] LNet: Removed LNI 192.168.206.108@tcp127 [11885.023127] LNet: Removed LNI 192.168.206.108@tcp128 [11886.047125] LNet: Removed LNI 192.168.206.108@tcp129 [11887.071124] LNet: Removed LNI 192.168.206.108@tcp130 [11888.095131] LNet: Removed LNI 192.168.206.108@tcp131 [11889.119229] LNet: Removed LNI 192.168.206.108@tcp132 [11890.143142] LNet: Removed LNI 192.168.206.108@tcp133 [11890.144941] LNet: Removed LNI 192.168.206.108@tcp134 [11891.173410] LNet: Removed LNI 192.168.206.108@tcp135 [11891.443422] Key type .llcrypt unregistered [11891.444403] Key type ._llcrypt unregistered [11898.616679] Key type ._llcrypt registered [11898.617805] Key type .llcrypt registered [11898.652365] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [11904.868452] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11905.187128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11905.217076] alg: No test for adler32 (adler32-zlib) [11906.065732] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [11906.150920] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [11907.727194] Key type lgssc registered [11908.096077] Lustre: Echo OBD driver; http://www.lustre.org/ [11919.128679] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:31:18 (1790958678) [11925.178870] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [11926.208841] Key type lgssc unregistered [11926.338271] LNet: 472453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11926.340507] LNetError: 472453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11926.350254] LNet: Removed LNI 192.168.206.108@tcp [11926.653336] Key type .llcrypt unregistered [11926.654402] Key type ._llcrypt unregistered [11933.371773] Key type ._llcrypt registered [11933.373202] Key type .llcrypt registered [11933.410536] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [11933.778539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11933.826793] alg: No test for adler32 (adler32-zlib) [11934.690351] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [11934.793602] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [11936.375199] Key type lgssc registered [11936.788698] Lustre: Echo OBD driver; http://www.lustre.org/ [11939.714438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11939.721506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11940.803763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11940.813831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11940.846126] Lustre: lustre-MDT0000: new disk, initializing [11940.865136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11940.870504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11942.179801] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11946.139770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11946.186118] Lustre: 473657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11946.210246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11946.211802] Lustre: Skipped 1 previous similar message [11946.244597] Lustre: lustre-MDT0001: new disk, initializing [11946.276388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11946.288983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11946.292706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11947.564132] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11951.126164] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11952.877901] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11954.976756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11955.070833] Lustre: lustre-OST0000: new disk, initializing [11955.072758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11955.075888] Lustre: 475622:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11955.103393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11956.934538] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [11959.093219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11959.097110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11959.127875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11960.458069] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11961.560916] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing lnet_if_list [11963.060436] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11963.069390] LNet: Added LNI 192.168.206.108@tcp10 [8/256/0/180] [11964.762822] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11964.774502] LNet: Added LNI 192.168.206.108@tcp11 [8/256/0/180] [11966.031966] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11966.043576] LNet: Added LNI 192.168.206.108@tcp12 [8/256/0/180] [11967.369367] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11967.381662] LNet: Added LNI 192.168.206.108@tcp13 [8/256/0/180] [11968.717468] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11968.726710] LNet: Added LNI 192.168.206.108@tcp14 [8/256/0/180] [11970.005538] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11970.015420] LNet: Added LNI 192.168.206.108@tcp15 [8/256/0/180] [11971.389810] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11971.398829] LNet: Added LNI 192.168.206.108@tcp16 [8/256/0/180] [11972.730263] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11972.740288] LNet: Added LNI 192.168.206.108@tcp17 [8/256/0/180] [11974.136413] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11974.148806] LNet: Added LNI 192.168.206.108@tcp18 [8/256/0/180] [11975.626737] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11975.636839] LNet: Added LNI 192.168.206.108@tcp19 [8/256/0/180] [11977.124295] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11977.135375] LNet: Added LNI 192.168.206.108@tcp20 [8/256/0/180] [11978.586941] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11978.598309] LNet: Added LNI 192.168.206.108@tcp21 [8/256/0/180] [11980.080612] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11980.090930] LNet: Added LNI 192.168.206.108@tcp22 [8/256/0/180] [11981.637347] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11981.646498] LNet: Added LNI 192.168.206.108@tcp23 [8/256/0/180] [11983.212907] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11983.223903] LNet: Added LNI 192.168.206.108@tcp24 [8/256/0/180] [11984.788568] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11984.799596] LNet: Added LNI 192.168.206.108@tcp25 [8/256/0/180] [11986.494676] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11986.504661] LNet: Added LNI 192.168.206.108@tcp26 [8/256/0/180] [11988.170244] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11988.179409] LNet: Added LNI 192.168.206.108@tcp27 [8/256/0/180] [11989.900827] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11989.912852] LNet: Added LNI 192.168.206.108@tcp28 [8/256/0/180] [11991.687934] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11991.700881] LNet: Added LNI 192.168.206.108@tcp29 [8/256/0/180] [11993.506036] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11993.516880] LNet: Added LNI 192.168.206.108@tcp30 [8/256/0/180] [11995.390333] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11995.401635] LNet: Added LNI 192.168.206.108@tcp31 [8/256/0/180] [11997.246425] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11997.257945] LNet: Added LNI 192.168.206.108@tcp32 [8/256/0/180] [11999.045564] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11999.057834] LNet: Added LNI 192.168.206.108@tcp33 [8/256/0/180] [12000.830590] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12000.840073] LNet: Added LNI 192.168.206.108@tcp34 [8/256/0/180] [12002.744705] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12002.754337] LNet: Added LNI 192.168.206.108@tcp35 [8/256/0/180] [12004.671166] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12004.683413] LNet: Added LNI 192.168.206.108@tcp36 [8/256/0/180] [12006.661186] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12006.670986] LNet: Added LNI 192.168.206.108@tcp37 [8/256/0/180] [12008.642267] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12008.651873] LNet: Added LNI 192.168.206.108@tcp38 [8/256/0/180] [12010.723062] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12010.731846] LNet: Added LNI 192.168.206.108@tcp39 [8/256/0/180] [12012.778215] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12012.788715] LNet: Added LNI 192.168.206.108@tcp40 [8/256/0/180] [12014.932701] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12014.941615] LNet: Added LNI 192.168.206.108@tcp41 [8/256/0/180] [12017.039825] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12017.049283] LNet: Added LNI 192.168.206.108@tcp42 [8/256/0/180] [12019.096225] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12019.105442] LNet: Added LNI 192.168.206.108@tcp43 [8/256/0/180] [12021.312242] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12021.322807] LNet: Added LNI 192.168.206.108@tcp44 [8/256/0/180] [12023.573874] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12023.582986] LNet: Added LNI 192.168.206.108@tcp45 [8/256/0/180] [12025.785623] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12025.797661] LNet: Added LNI 192.168.206.108@tcp46 [8/256/0/180] [12028.322479] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12028.334292] LNet: Added LNI 192.168.206.108@tcp47 [8/256/0/180] [12030.864231] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12030.873301] LNet: Added LNI 192.168.206.108@tcp48 [8/256/0/180] [12033.157199] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12033.166243] LNet: Added LNI 192.168.206.108@tcp49 [8/256/0/180] [12035.429169] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12035.438608] LNet: Added LNI 192.168.206.108@tcp50 [8/256/0/180] [12043.743766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12043.746093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12043.749985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.304225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.306026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12048.605437] Lustre: server umount lustre-MDT0000 complete [12049.842051] LustreError: 491728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958809 with bad export cookie 6428687928967249260 [12049.845727] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12049.845951] LustreError: 491728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12049.954616] Lustre: server umount lustre-MDT0001 complete [12057.425263] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [12060.362376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12067.807284] Lustre: 472922:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790958811/real 1790958811] req@00000000b4a1bf85 x1877956303598720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790958827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12067.814696] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.818456] Lustre: Skipped 1 previous similar message [12070.815100] Lustre: 472923:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790958814/real 1790958814] req@0000000063586850 x1877956303598848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790958830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12072.927163] Lustre: 472920:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790958816/real 1790958816] req@0000000074b36fdf x1877956303598976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790958832 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12073.951454] LustreError: 472919:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b5e1d985 x1877956303599872/t0(0) o250->MGC192.168.206.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12074.066344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12075.103081] Lustre: 472920:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790958819/real 1790958819] req@00000000e5e91e32 x1877956303599232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790958835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12075.478403] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12078.459482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12078.593784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12080.053593] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12080.095086] Lustre: 472922:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790958824/real 1790958824] req@00000000b17f4fcf x1877956303599360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790958840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12082.603028] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12083.685021] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12084.374332] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12086.756576] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12091.872933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12091.873962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12091.876509] Lustre: Skipped 1 previous similar message [12091.879304] Lustre: Skipped 3 previous similar messages [12096.992901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12096.995234] Lustre: Skipped 1 previous similar message [12097.560740] Lustre: server umount lustre-OST0000 complete [12099.039760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12099.042021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12102.114218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12102.116720] Lustre: Skipped 2 previous similar messages [12112.352264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12112.354583] Lustre: Skipped 3 previous similar messages [12113.375106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12113.436738] Lustre: server umount lustre-MDT0000 complete [12114.687330] LustreError: 473649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958874 with bad export cookie 6428687928967251283 [12114.691641] LustreError: 473649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12114.693226] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12114.820449] Lustre: server umount lustre-MDT0001 complete [12118.500263] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [12119.568331] Key type lgssc unregistered [12119.718167] LNet: 502582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12119.721341] LNetError: 502582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12119.723808] LNet: Removed LNI 192.168.206.108@tcp [12119.725742] LNet: Removed LNI 192.168.206.108@tcp10 [12119.728072] LNet: Removed LNI 192.168.206.108@tcp11 [12119.729778] LNet: Removed LNI 192.168.206.108@tcp12 [12119.730850] LNet: Removed LNI 192.168.206.108@tcp13 [12119.732537] LNet: Removed LNI 192.168.206.108@tcp14 [12119.734800] LNet: Removed LNI 192.168.206.108@tcp15 [12119.737074] LNet: Removed LNI 192.168.206.108@tcp16 [12119.739053] LNet: Removed LNI 192.168.206.108@tcp17 [12119.741398] LNet: Removed LNI 192.168.206.108@tcp18 [12119.743478] LNet: Removed LNI 192.168.206.108@tcp19 [12119.745542] LNet: Removed LNI 192.168.206.108@tcp20 [12119.747803] LNet: Removed LNI 192.168.206.108@tcp21 [12119.750082] LNet: Removed LNI 192.168.206.108@tcp22 [12119.751831] LNet: Removed LNI 192.168.206.108@tcp23 [12119.753075] LNet: Removed LNI 192.168.206.108@tcp24 [12119.754169] LNet: Removed LNI 192.168.206.108@tcp25 [12119.755288] LNet: Removed LNI 192.168.206.108@tcp26 [12119.756402] LNet: Removed LNI 192.168.206.108@tcp27 [12119.757785] LNet: Removed LNI 192.168.206.108@tcp28 [12119.759824] LNet: Removed LNI 192.168.206.108@tcp29 [12119.762054] LNet: Removed LNI 192.168.206.108@tcp30 [12119.763911] LNet: Removed LNI 192.168.206.108@tcp31 [12119.766127] LNet: Removed LNI 192.168.206.108@tcp32 [12119.768353] LNet: Removed LNI 192.168.206.108@tcp33 [12119.770586] LNet: Removed LNI 192.168.206.108@tcp34 [12119.772798] LNet: Removed LNI 192.168.206.108@tcp35 [12119.775050] LNet: Removed LNI 192.168.206.108@tcp36 [12119.777179] LNet: Removed LNI 192.168.206.108@tcp37 [12119.779427] LNet: Removed LNI 192.168.206.108@tcp38 [12119.781627] LNet: Removed LNI 192.168.206.108@tcp39 [12119.783875] LNet: Removed LNI 192.168.206.108@tcp40 [12119.786108] LNet: Removed LNI 192.168.206.108@tcp41 [12119.788302] LNet: Removed LNI 192.168.206.108@tcp42 [12119.790554] LNet: Removed LNI 192.168.206.108@tcp43 [12119.792706] LNet: Removed LNI 192.168.206.108@tcp44 [12119.794886] LNet: Removed LNI 192.168.206.108@tcp45 [12119.797074] LNet: Removed LNI 192.168.206.108@tcp46 [12119.799287] LNet: Removed LNI 192.168.206.108@tcp47 [12119.801465] LNet: Removed LNI 192.168.206.108@tcp48 [12119.803668] LNet: Removed LNI 192.168.206.108@tcp49 [12119.810357] LNet: Removed LNI 192.168.206.108@tcp50 [12120.077904] Key type .llcrypt unregistered [12120.079169] Key type ._llcrypt unregistered [12126.636654] Key type ._llcrypt registered [12126.637721] Key type .llcrypt registered [12126.673143] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [12132.940031] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [12133.290806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12133.322382] alg: No test for adler32 (adler32-zlib) [12134.173249] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [12134.258308] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [12135.839160] Key type lgssc registered [12136.224587] Lustre: Echo OBD driver; http://www.lustre.org/ [12147.563698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12147.570449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12148.654897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12148.664559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12148.703493] Lustre: lustre-MDT0000: new disk, initializing [12148.725467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12148.731615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12150.047667] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12154.097504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12154.139752] Lustre: 505764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12154.164959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12154.167236] Lustre: Skipped 1 previous similar message [12154.202739] Lustre: lustre-MDT0001: new disk, initializing [12154.234929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12154.247100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12154.250244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12155.566555] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12159.385902] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12161.282523] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12163.986558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12164.067667] Lustre: *** cfs_fail_loc=90f, val=0*** [12164.106766] Lustre: lustre-OST0000: new disk, initializing [12164.109875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12164.114407] Lustre: 507778:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12164.150716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12166.101885] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12169.850074] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12172.414137] Lustre: server umount lustre-OST0000 complete [12173.809270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12173.810361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12173.814493] Lustre: Skipped 1 previous similar message [12178.912428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12178.915571] Lustre: Skipped 2 previous similar messages [12179.802034] Lustre: server umount lustre-MDT0000 complete [12181.075388] LustreError: 505757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790958941 with bad export cookie 4439862489727910760 [12181.078043] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12181.080370] LustreError: 505757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12181.200731] Lustre: server umount lustre-MDT0001 complete [12185.143811] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [12186.283934] Key type lgssc unregistered [12186.426507] LNet: 509674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12186.429757] LNetError: 509674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12186.439427] LNet: Removed LNI 192.168.206.108@tcp [12186.782512] Key type .llcrypt unregistered [12186.784294] Key type ._llcrypt unregistered [12194.538958] Key type ._llcrypt registered [12194.540493] Key type .llcrypt registered [12194.596553] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [12195.178496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12195.197494] alg: No test for adler32 (adler32-zlib) [12196.074801] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [12196.184971] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [12197.775151] Key type lgssc registered [12198.167762] Lustre: Echo OBD driver; http://www.lustre.org/ [12200.757753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12200.765833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12201.911229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12203.255249] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12206.044559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.177972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12207.544683] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12209.946244] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12211.785605] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12214.024055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12214.144759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12216.060709] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12216.230611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12216.234385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12218.464523] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12226.480095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12232.545308] loop0: detected capacity change from 0 to 204800000 [12232.596222] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12232.601761] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12232.607531] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12234.562352] loop0: detected capacity change from 0 to 204800000 [12234.626086] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12234.630427] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12234.635381] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12236.272288] loop0: detected capacity change from 0 to 204800000 [12238.091805] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12238.147291] Lustre: fs15246-OST0000: new disk, initializing [12238.149667] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12238.153030] Lustre: 514548:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12238.171385] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12240.075656] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12243.859745] loop1: detected capacity change from 0 to 204800000 [12245.648863] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12245.688099] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12245.710173] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12245.749682] Lustre: fs15246-MDT0000: new disk, initializing [12245.796367] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12245.804465] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12245.845386] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12246.879728] Lustre: 515669:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000bc0b04e5 x1877956578277888/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790959022 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12247.108053] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12255.967298] LustreError: 515669:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [12255.972379] LustreError: 515669:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [12255.975326] LustreError: 515669:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [12257.762701] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12257.766705] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12260.377687] Lustre: server umount fs15246-MDT0000 complete [12262.882111] LustreError: 510929:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12262.888244] LustreError: 510929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12263.396826] Lustre: server umount fs15246-OST0000 complete [12267.487952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12267.488619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.490130] LustreError: Skipped 1 previous similar message [12267.494467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12267.498098] Lustre: Skipped 1 previous similar message [12273.122600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12278.242573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12278.244426] Lustre: Skipped 2 previous similar messages [12281.311135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12281.366882] Lustre: server umount lustre-OST0000 complete [12282.847713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12282.850153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12282.853205] Lustre: Skipped 1 previous similar message [12282.854651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12288.664791] Lustre: server umount lustre-MDT0000 complete [12289.905540] LustreError: 510915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959049 with bad export cookie 12394728312035513221 [12289.908127] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12289.908991] LustreError: 510915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12290.036944] Lustre: server umount lustre-MDT0001 complete [12293.665203] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [12294.639757] Key type lgssc unregistered [12294.767369] LNet: 518763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12294.769914] LNetError: 518763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12294.779301] LNet: Removed LNI 192.168.206.108@tcp [12295.093394] Key type .llcrypt unregistered [12295.094346] Key type ._llcrypt unregistered [12302.507304] Key type ._llcrypt registered [12302.508366] Key type .llcrypt registered [12302.548677] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [12302.940701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12302.991436] alg: No test for adler32 (adler32-zlib) [12303.858198] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [12303.959299] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [12305.543149] Key type lgssc registered [12305.916482] Lustre: Echo OBD driver; http://www.lustre.org/ [12318.372146] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [12321.600964] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [12333.160967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12333.167377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12334.253188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12334.264795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12334.301621] Lustre: lustre-MDT0000: new disk, initializing [12334.323882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12334.329159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12335.696596] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12339.769064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12339.814027] Lustre: 523213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12339.840925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12339.842937] Lustre: Skipped 1 previous similar message [12339.879311] Lustre: lustre-MDT0001: new disk, initializing [12339.916353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12339.930648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12339.935863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12341.286175] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12345.094221] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12346.921827] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12349.263759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12349.390434] Lustre: lustre-OST0000: new disk, initializing [12349.392569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12349.395320] Lustre: 525182:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12349.432305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12351.371463] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [12355.065395] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12356.590233] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12359.680752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12359.685338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12359.722849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12360.752901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [12362.093945] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12362.182440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12364.769122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12364.770778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12364.771320] LustreError: Skipped 1 previous similar message [12364.775164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12364.778368] Lustre: Skipped 1 previous similar message [12368.918682] Lustre: server umount lustre-OST0000 complete [12370.911791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12370.913912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12370.916849] Lustre: Skipped 1 previous similar message [12370.918423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12375.008400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12375.008865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12376.343349] Lustre: server umount lustre-MDT0000 complete [12377.539489] LustreError: 523207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790959137 with bad export cookie 10983688779235923445 [12377.542756] LustreError: 523207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12377.543697] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12377.658494] Lustre: server umount lustre-MDT0001 complete [12385.448990] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [12388.726472] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [12399.149746] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12130 sec ========= 12:39:18 (1790959158) [12399.729403] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:39:19 (1790959159) === [12400.955299] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:39:20 (1790959160) === [12414.446938] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [12415.533210] Key type lgssc unregistered [12415.666333] LNet: 531840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12415.668909] LNetError: 531840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12415.681271] LNet: Removed LNI 192.168.206.108@tcp [12415.999664] Key type .llcrypt unregistered [12416.000873] Key type ._llcrypt unregistered