[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475934766 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002341] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009109] LSM: Security Framework initializing [ 0.010039] Yama: becoming mindful. [ 0.011050] SELinux: Initializing. [ 0.012050] *** VALIDATE selinux *** [ 0.019469] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023594] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025067] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] *** VALIDATE tmpfs *** [ 0.029439] *** VALIDATE proc *** [ 0.031034] *** VALIDATE cgroup *** [ 0.031764] *** VALIDATE cgroup2 *** [ 0.032208] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035022] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.039539] debug: unmapping init [mem 0xffffffffb0259000-0xffffffffb0260fff] [ 0.041162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042608] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043022] ... version: 2 [ 0.043889] ... bit width: 48 [ 0.044014] ... generic registers: 4 [ 0.044795] ... value mask: 0000ffffffffffff [ 0.045012] ... max period: 00007fffffffffff [ 0.046014] ... fixed-purpose events: 3 [ 0.046956] ... event mask: 000000070000000f [ 0.048217] rcu: Hierarchical SRCU implementation. [ 0.050312] smp: Bringing up secondary CPUs ... [ 0.051480] x86: Booting SMP configuration: [ 0.052022] .... node #0, CPUs: #1 #2 #3 [ 0.055660] smp: Brought up 1 node, 4 CPUs [ 0.057012] smpboot: Max logical packages: 1 [ 0.058019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.239383] node 0 deferred pages initialised in 178ms [ 0.243090] devtmpfs: initialized [ 0.244333] x86/mm: Memory block size: 128MB [ 0.248175] gcov: version magic: 0x41383552 [ 0.252456] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.256136] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.260458] pinctrl core: initialized pinctrl subsystem [ 0.262413] [ 0.264012] ************************************************************* [ 0.267022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.271022] ** ** [ 0.273037] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.276026] ** ** [ 0.279019] ** This means that this kernel is built to expose internal ** [ 0.282020] ** IOMMU data structures, which may compromise security on ** [ 0.284020] ** your system. ** [ 0.287019] ** ** [ 0.289031] ** If you see this message and you are not debugging the ** [ 0.292021] ** kernel, report this immediately to your vendor! ** [ 0.294020] ** ** [ 0.296022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.299023] ************************************************************* [ 0.302691] NET: Registered protocol family 16 [ 0.305451] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.307066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.310066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.313161] cpuidle: using governor menu [ 0.314613] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.315775] PCI: Using configuration type 1 for base access [ 0.317170] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.327101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.330062] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.334180] cryptd: max_cpu_qlen set to 1000 [ 0.338344] ACPI: Added _OSI(Module Device) [ 0.340023] ACPI: Added _OSI(Processor Device) [ 0.342021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.344018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349141] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.357259] ACPI: Interpreter enabled [ 0.358090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.360018] ACPI: Using IOAPIC for interrupt routing [ 0.362163] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.366500] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.377967] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.380068] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.383030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.387131] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.393586] acpiphp: Slot [2] registered [ 0.395217] acpiphp: Slot [5] registered [ 0.397223] acpiphp: Slot [6] registered [ 0.399246] acpiphp: Slot [7] registered [ 0.402269] acpiphp: Slot [8] registered [ 0.404150] acpiphp: Slot [9] registered [ 0.406220] acpiphp: Slot [10] registered [ 0.407184] acpiphp: Slot [3] registered [ 0.409114] acpiphp: Slot [4] registered [ 0.411183] acpiphp: Slot [11] registered [ 0.413181] acpiphp: Slot [12] registered [ 0.415168] acpiphp: Slot [13] registered [ 0.417135] acpiphp: Slot [14] registered [ 0.418134] acpiphp: Slot [15] registered [ 0.420203] acpiphp: Slot [16] registered [ 0.422113] acpiphp: Slot [17] registered [ 0.424146] acpiphp: Slot [18] registered [ 0.425128] acpiphp: Slot [19] registered [ 0.427139] acpiphp: Slot [20] registered [ 0.429165] acpiphp: Slot [21] registered [ 0.431132] acpiphp: Slot [22] registered [ 0.432114] acpiphp: Slot [23] registered [ 0.434125] acpiphp: Slot [24] registered [ 0.435108] acpiphp: Slot [25] registered [ 0.437131] acpiphp: Slot [26] registered [ 0.438089] acpiphp: Slot [27] registered [ 0.440099] acpiphp: Slot [28] registered [ 0.441174] acpiphp: Slot [29] registered [ 0.443128] acpiphp: Slot [30] registered [ 0.444096] acpiphp: Slot [31] registered [ 0.446113] PCI host bridge to bus 0000:00 [ 0.447048] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.450041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.451029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.455047] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.459050] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.462042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.467253] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.471291] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.474964] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.484777] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.489046] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.491025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.493028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.496030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.499710] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.503271] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.506048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.508983] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.514017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.529033] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.533908] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.541255] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.548021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.557028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.581021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.591297] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.600024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.609017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.629016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.640000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.647028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.666020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.683025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.694135] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.702019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.712019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.732021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.745436] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.752025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.759022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.773015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.783452] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.790021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.795015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.812017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.827968] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.830456] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.832386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.834514] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.837460] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.842118] iommu: Default domain type: Passthrough [ 0.843452] SCSI subsystem initialized [ 0.845179] ACPI: bus type USB registered [ 0.847134] usbcore: registered new interface driver usbfs [ 0.849119] usbcore: registered new interface driver hub [ 0.854106] usbcore: registered new device driver usb [ 0.856218] pps_core: LinuxPPS API ver. 1 registered [ 0.858013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.861072] PTP clock support registered [ 0.864065] EDAC MC: Ver: 3.0.0 [ 0.866177] PCI: Using ACPI for IRQ routing [ 0.868589] NetLabel: Initializing [ 0.870011] NetLabel: domain hash size = 128 [ 0.871010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.873084] NetLabel: unlabeled traffic allowed by default [ 0.874123] vgaarb: loaded [ 0.876400] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.877012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.882746] clocksource: Switched to clocksource kvm-clock [ 0.987431] VFS: Disk quotas dquot_6.6.0 [ 0.989370] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.992308] *** VALIDATE ramfs *** [ 0.993533] *** VALIDATE hugetlbfs *** [ 0.995668] pnp: PnP ACPI init [ 0.998285] pnp: PnP ACPI: found 6 devices [ 1.021712] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.025453] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.028019] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.030426] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.033304] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.035764] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.038577] NET: Registered protocol family 2 [ 1.041391] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.046567] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.050339] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.055706] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.059561] TCP: Hash tables configured (established 65536 bind 65536) [ 1.062694] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.065949] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.069236] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.072954] NET: Registered protocol family 1 [ 1.076079] RPC: Registered named UNIX socket transport module. [ 1.078247] RPC: Registered udp transport module. [ 1.079567] RPC: Registered tcp transport module. [ 1.080986] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.082916] NET: Registered protocol family 44 [ 1.085086] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.087359] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.089655] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.092300] PCI: CLS 0 bytes, default 64 [ 1.094623] Unpacking initramfs... [ 2.529688] debug: unmapping init [mem 0xffff9135bcc54000-0xffff9135bffbffff] [ 2.534717] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.538734] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.542053] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.033099] Initialise system trusted keyrings [ 3.034506] Key type blacklist registered [ 3.036224] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.045703] zbud: loaded [ 3.048956] *** VALIDATE nfs *** [ 3.050299] *** VALIDATE nfs4 *** [ 3.051976] pstore: using deflate compression [ 3.055073] Platform Keyring initialized [ 3.148539] NET: Registered protocol family 38 [ 3.151047] Key type asymmetric registered [ 3.152945] Asymmetric key parser 'x509' registered [ 3.154880] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.158265] io scheduler mq-deadline registered [ 3.160149] io scheduler kyber registered [ 3.161811] io scheduler bfq registered [ 3.164046] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.166614] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.169363] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.172276] ACPI: Power Button [PWRF] [ 3.178258] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.184969] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.195220] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.201446] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.213901] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.241764] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.270760] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.278324] Non-volatile memory driver v1.3 [ 3.280097] Linux agpgart interface v0.103 [ 3.309771] virtio_blk virtio1: [vda] 133944 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.312975] vda: detected capacity change from 0 to 68579328 [ 3.327472] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.330656] vdb: detected capacity change from 0 to 1073741824 [ 3.346634] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.351471] vdc: detected capacity change from 0 to 2621440000 [ 3.365804] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.368813] vdd: detected capacity change from 0 to 2621440000 [ 3.385989] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.389761] vde: detected capacity change from 0 to 4294967296 [ 3.405641] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.408993] vdf: detected capacity change from 0 to 4294967296 [ 3.416285] libphy: Fixed MDIO Bus: probed [ 3.422942] usbcore: registered new interface driver usbserial_generic [ 3.426449] usbserial: USB Serial support registered for generic [ 3.429440] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.434493] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.451188] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.455589] mousedev: PS/2 mouse device common for all mice [ 3.459709] rtc_cmos 00:05: RTC can wake from S4 [ 3.463418] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.465715] rtc_cmos 00:05: registered as rtc0 [ 3.469700] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.471032] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.475040] intel_pstate: CPU model not supported [ 3.480107] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.485290] hid: raw HID events driver (C) Jiri Kosina [ 3.487892] usbcore: registered new interface driver usbhid [ 3.490153] usbhid: USB HID core driver [ 3.493431] drop_monitor: Initializing network drop monitor service [ 3.496255] Initializing XFRM netlink socket [ 3.498383] NET: Registered protocol family 10 [ 3.501445] Segment Routing with IPv6 [ 3.502573] NET: Registered protocol family 17 [ 3.504243] mpls_gso: MPLS GSO support [ 3.510610] RAS: Correctable Errors collector initialized. [ 3.512758] AVX version of gcm_enc/dec engaged. [ 3.513779] AES CTR mode by8 optimization enabled [ 3.585867] sched_clock: Marking stable (3585836964, 0)->(4463880081, -878043117) [ 3.588863] registered taskstats version 1 [ 3.590427] Loading compiled-in X.509 certificates [ 3.592122] zswap: loaded using pool lzo/zbud [ 3.614140] Key type big_key registered [ 3.625517] Key type encrypted registered [ 3.627590] ima: No TPM chip found, activating TPM-bypass! [ 3.629561] ima: Allocated hash algorithm: sha1 [ 3.631873] ima: No architecture policies found [ 3.633874] evm: Initialising EVM extended attributes: [ 3.635973] evm: security.selinux [ 3.637470] evm: security.ima [ 3.638439] evm: security.capability [ 3.639607] evm: HMAC attrs: 0x1 [ 3.642309] rtc_cmos 00:05: setting system clock to 2026-04-03 05:26:05 UTC (1775193965) [ 3.649239] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 3.652221] debug: unmapping init [mem 0xffffffffaff82000-0xffffffffb0258fff] [ 3.657089] Write protecting the kernel read-only data: 28672k [ 3.660741] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 3.663876] debug: unmapping init [mem 0xffffffffaef14000-0xffffffffaeffffff] [ 3.695394] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.704690] systemd[1]: Detected virtualization kvm. [ 3.706538] systemd[1]: Detected architecture x86-64. [ 3.708569] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.732259] systemd[1]: No hostname configured. [ 3.734206] systemd[1]: Set hostname to . [ 3.736498] random: systemd: uninitialized urandom read (16 bytes read) [ 3.739010] systemd[1]: Initializing machine ID from random generator. [ 3.784988] random: ln: uninitialized urandom read (6 bytes read) [ 3.873821] random: systemd: uninitialized urandom read (16 bytes read) [ 3.877175] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.883763] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.889679] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.508517] device-mapper: uevent: version 1.0.3 [ 4.510393] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.274784] virtio_net virtio0 ens2: renamed from eth0 [ 5.279083] random: fast init done [ 5.306365] scsi host0: ata_piix [ 5.340101] scsi host1: ata_piix [ 5.342069] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.345144] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.156429] random: crng init done [ 10.162274] random: 7 urandom warning(s) missed due to ratelimiting [ 10.169062] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.265394] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.812641] printk: systemd: 25 output lines suppressed due to ratelimiting [ 16.743784] SELinux: Disabled at runtime. [ 16.891364] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.914818] systemd[1]: Detected virtualization kvm. [ 16.916830] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.729434] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.744302] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.754414] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.765564] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.776386] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.808582] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.834325] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ 19.431144] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.228775] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.133477] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.200084] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.685881] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.807107] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.791042] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.667572] NFS: Registering the id_resolver key type [ 27.670702] Key type id_resolver registered [ 27.674724] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg638-server login: [ 77.376244] libcfs: loading out-of-tree module taints kernel. [ 77.446910] Key type ._llcrypt registered [ 77.450863] Key type .llcrypt registered [ 77.575472] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 96.706567] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 98.874071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.894236] alg: No test for adler32 (adler32-zlib) [ 100.580066] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 101.867188] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 103.769438] Key type lgssc registered [ 105.937614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.820624] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 169.783676] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 181.850916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 181.888630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.151918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.196932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.260401] Lustre: lustre-MDT0000: new disk, initializing [ 183.350742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.394714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 186.898378] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.151201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.235403] Lustre: 6508:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.260979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 199.266925] Lustre: Skipped 1 previous similar message [ 199.357608] Lustre: lustre-MDT0001: new disk, initializing [ 199.479478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.518865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.528489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.632729] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.078301] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 216.272769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.437359] Lustre: lustre-OST0000: new disk, initializing [ 216.440280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 216.480638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.131351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.140957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.212073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.450831] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.017597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.161441] Lustre: lustre-OST0001: new disk, initializing [ 237.164507] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.197685] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.538377] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.305027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 247.308252] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 247.341727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 257.278248] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.259901] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.190088] hrtimer: interrupt took 2185858 ns [ 274.381841] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing check_logdir /tmp/testlogs/ [ 279.327369] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing yml_node [ 284.095170] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 286.389227] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 288.790964] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 290.888032] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 3 01:30:50 EDT 2026 [ 307.104445] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 308.332825] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 324.064161] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.068991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.088833] Lustre: Skipped 1 previous similar message [ 329.186982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.196085] Lustre: Skipped 6 previous similar messages [ 329.518710] Lustre: server umount lustre-MDT0000 complete [ 332.310745] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775194294 with bad export cookie 8573225310432525110 [ 332.311595] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 332.321461] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 334.304065] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.317736] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 334.330564] Lustre: Skipped 3 previous similar messages [ 334.341226] Lustre: Skipped 1 previous similar message [ 337.312031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 337.320979] Lustre: Skipped 1 previous similar message [ 338.693673] Lustre: server umount lustre-MDT0001 complete [ 342.179656] Lustre: server umount lustre-OST0000 complete [ 345.608109] Lustre: server umount lustre-OST0001 complete [ 351.761303] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 359.314082] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 393.039325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.352483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 393.402066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 393.552762] Lustre: lustre-MDT0000: new disk, initializing [ 393.730830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.776632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 398.307364] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 410.169955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.338542] Lustre: 14975:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 410.392092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 410.395157] Lustre: Skipped 1 previous similar message [ 410.455538] Lustre: lustre-MDT0001: new disk, initializing [ 410.530719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 410.573765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 410.581068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 414.264752] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 421.670454] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 425.520634] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 431.775936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 431.990308] Lustre: lustre-OST0000: new disk, initializing [ 431.998495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 432.098915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.322279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 433.333573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 433.434099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 437.857193] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 445.464412] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 449.960305] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 450.207683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 454.381265] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 454.559735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 459.240108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 459.242218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.243832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.244856] LustreError: Skipped 1 previous similar message [ 459.267477] Lustre: Skipped 1 previous similar message [ 462.190479] Lustre: server umount lustre-OST0000 complete [ 466.914538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.933910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.478742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 469.484126] Lustre: Skipped 3 previous similar messages [ 472.016650] Lustre: server umount lustre-MDT0000 complete [ 474.606873] LustreError: 14986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.638928] LustreError: 14986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 475.880525] LustreError: 14967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775194437 with bad export cookie 8573225310432526643 [ 475.882823] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.888163] LustreError: 14967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 476.270513] Lustre: server umount lustre-MDT0001 complete [ 480.473630] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:34:00 (1775194440) [ 482.710524] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 484.555857] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:34:04 (1775194444) [ 491.030266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.553418] LustreError: 18979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.651857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.139687] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 497.129582] LustreError: 18980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.223127] LustreError: 18979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.561707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.488419] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 515.579278] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 520.241575] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 528.239046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 534.772278] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 540.132118] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 557.536495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 557.547428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.573972] Lustre: Skipped 1 previous similar message [ 557.576604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 562.559702] Lustre: server umount lustre-OST0000 complete [ 568.801554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.827599] Lustre: Skipped 2 previous similar messages [ 573.777198] Lustre: server umount lustre-MDT0000 complete [ 573.921838] LustreError: 18980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.951050] LustreError: 18980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 578.736807] LustreError: 18959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775194540 with bad export cookie 8573225310432527525 [ 578.740855] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.760049] LustreError: 18959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 579.012104] Lustre: server umount lustre-MDT0001 complete [ 590.107329] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 593.292595] Key type lgssc unregistered [ 593.777635] LNet: 22183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.784116] LNetError: 22183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.817109] LNet: Removed LNI 192.168.206.138@tcp [ 594.708183] Key type .llcrypt unregistered [ 594.710460] Key type ._llcrypt unregistered [ 617.577792] Key type ._llcrypt registered [ 617.580265] Key type .llcrypt registered [ 617.684227] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 618.648393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 618.799680] alg: No test for adler32 (adler32-zlib) [ 619.836760] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 620.076178] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 621.737665] Key type lgssc registered [ 623.049949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.637082] loop: module loaded [ 631.756642] loop0: detected capacity change from 0 to 204800000 [ 631.995744] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.030252] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.219419] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.880842] loop0: detected capacity change from 0 to 204800000 [ 639.098113] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.111623] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.140094] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.065957] loop0: detected capacity change from 0 to 204800000 [ 649.721779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 649.749264] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.014090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 651.095147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 651.154822] Lustre: lustre-MDT0000: new disk, initializing [ 651.274916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.297076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 655.326581] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 665.402929] loop1: detected capacity change from 0 to 204800000 [ 670.789617] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 671.082241] Lustre: lustre-OST07c6: new disk, initializing [ 671.089680] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 671.097090] Lustre: Skipped 1 previous similar message [ 671.160300] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 678.151897] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 679.511642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 679.520808] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 679.555849] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 683.228147] Lustre: 24346:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 683.255816] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 683.271508] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 683.283980] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 683.306970] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 683.324814] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 701.298802] Lustre: 24345:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 701.308274] Lustre: 24345:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 701.317748] Lustre: 24345:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 701.327971] Lustre: 24345:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 701.341712] Lustre: 24345:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 701.351220] Lustre: 24345:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 706.441643] Lustre: 24345:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 706.469769] Lustre: 24345:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 706.481347] Lustre: 24345:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 706.493885] Lustre: 24345:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.501868] Lustre: 24345:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 706.514726] Lustre: 24345:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.530341] Lustre: 24345:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 706.541866] Lustre: 24345:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.560608] Lustre: 24345:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 706.569316] Lustre: 24345:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.575538] Lustre: 24345:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 706.581856] Lustre: 24345:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.759256] Lustre: 24347:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 709.770496] Lustre: 24347:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 709.775684] Lustre: 24347:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 709.782975] Lustre: 24347:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.787797] Lustre: 24347:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 709.793704] Lustre: 24347:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.800249] Lustre: 24347:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 709.807773] Lustre: 24347:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.815459] Lustre: 24347:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 709.827538] Lustre: 24347:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.838142] Lustre: 24347:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 709.843710] Lustre: 24347:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.745845] Lustre: 24345:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 714.749870] Lustre: 24345:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 714.753366] Lustre: 24345:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 714.757687] Lustre: 24345:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.761923] Lustre: 24345:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 714.766200] Lustre: 24345:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.769595] Lustre: 24345:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 714.777249] Lustre: 24345:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.783784] Lustre: 24345:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 714.789366] Lustre: 24345:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.797802] Lustre: 24345:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 714.803840] Lustre: 24345:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.185381] Lustre: 24346:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 723.206138] Lustre: 24346:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 3 previous similar messages [ 723.221722] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 723.238228] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 723.258869] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 723.271200] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 723.284554] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 723.302547] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 723.311688] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 723.331275] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 723.352704] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 723.364163] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 776.672710] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.695085] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 777.376883] Lustre: server umount lustre-OST07c6 complete [ 786.344114] Lustre: server umount lustre-MDT0000 complete [ 803.035156] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 805.853832] Key type lgssc unregistered [ 806.033786] LNet: 27868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.039273] LNetError: 27868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.086223] LNet: Removed LNI 192.168.206.138@tcp [ 806.809924] Key type .llcrypt unregistered [ 806.811363] Key type ._llcrypt unregistered [ 828.499988] Key type ._llcrypt registered [ 828.502265] Key type .llcrypt registered [ 828.591353] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 829.623256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 829.714513] alg: No test for adler32 (adler32-zlib) [ 831.004838] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 831.398173] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 833.128556] Key type lgssc registered [ 834.472886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 842.273912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 842.291887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.865444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.036862] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 853.998076] LustreError: 29123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 854.031167] LustreError: 29123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 855.418664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.760816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 859.554943] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 865.661441] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 870.145978] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 877.308555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.677984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.978405] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 888.948295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 889.504394] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 904.676512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.702803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 904.705672] Lustre: Skipped 1 previous similar message [ 909.797235] LustreError: 30719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 909.814662] LustreError: 30719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 909.913607] Lustre: server umount lustre-OST0000 complete [ 914.914515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.926560] Lustre: Skipped 1 previous similar message [ 914.939413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 920.033411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 920.045578] Lustre: Skipped 2 previous similar messages [ 925.157987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.223469] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 928.396441] Lustre: server umount lustre-MDT0000 complete [ 930.273663] LustreError: 29123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.291409] LustreError: 29123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 932.274636] LustreError: 29103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775194894 with bad export cookie 1056005383942432075 [ 932.276560] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 932.282946] LustreError: 29103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 932.803947] Lustre: server umount lustre-MDT0001 complete [ 944.130742] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 946.903895] Key type lgssc unregistered [ 947.232476] LNet: 32512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.238910] LNetError: 32512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.253181] LNet: Removed LNI 192.168.206.138@tcp [ 948.181979] Key type .llcrypt unregistered [ 948.186123] Key type ._llcrypt unregistered [ 965.739109] Key type ._llcrypt registered [ 965.745988] Key type .llcrypt registered [ 965.826175] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 966.538305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.591041] alg: No test for adler32 (adler32-zlib) [ 967.671880] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 967.869523] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 969.543435] Key type lgssc registered [ 970.421959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.520875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.549938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.203960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 984.633199] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 990.695288] LustreError: 33765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.718790] LustreError: 33765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 993.401730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.869721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 998.603461] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1005.873062] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1010.975480] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1018.172029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.350858] LustreError: 35360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.682427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.931728] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1026.050810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1032.429170] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1044.448499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1044.454894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.469979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1046.498438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1046.507388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.510895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1048.854949] Lustre: server umount lustre-OST0000 complete [ 1055.203833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1055.211693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.222493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.226043] Lustre: Skipped 1 previous similar message [ 1058.850435] Lustre: server umount lustre-MDT0000 complete [ 1061.859882] LustreError: 33761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.891181] LustreError: 33761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1062.260157] LustreError: 33745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775195024 with bad export cookie 17890231125274847715 [ 1062.265578] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.275908] LustreError: 33745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1062.609917] Lustre: server umount lustre-MDT0001 complete [ 1072.195427] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1075.136973] Key type lgssc unregistered [ 1075.417133] LNet: 36947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.425692] LNetError: 36947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.439052] LNet: Removed LNI 192.168.206.138@tcp [ 1076.169736] Key type .llcrypt unregistered [ 1076.173248] Key type ._llcrypt unregistered [ 1101.351395] Key type ._llcrypt registered [ 1101.352845] Key type .llcrypt registered [ 1101.491682] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1102.615284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.654279] alg: No test for adler32 (adler32-zlib) [ 1103.740562] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 1104.077505] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1105.791153] Key type lgssc registered [ 1106.997525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1115.723269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1115.752698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.308056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.492572] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1127.911824] LustreError: 38348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.940763] LustreError: 38348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1132.299921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.828230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1137.839608] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1143.924762] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1147.788913] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1154.391804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1154.714732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.279799] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1162.944118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1166.878744] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1173.479994] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.488741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1173.503302] Lustre: Skipped 1 previous similar message [ 1173.530587] Lustre: Skipped 2 previous similar messages [ 1174.599127] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1175.602720] Lustre: server umount lustre-MDT0000 complete [ 1178.608345] LustreError: 39059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.628100] LustreError: 39059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1179.700951] LustreError: 38344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.908371] LustreError: 38329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775195141 with bad export cookie 2400292547832379448 [ 1179.924809] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.713863] LustreError: 39961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.715108] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.753539] Lustre: Skipped 1 previous similar message [ 1183.760084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1186.364443] Lustre: server umount lustre-MDT0001 complete [ 1207.226749] Lustre: server umount lustre-OST0000 complete [ 1223.494688] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1226.723139] Key type lgssc unregistered [ 1227.029133] LNet: 41856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.046422] LNetError: 41856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.068269] LNet: Removed LNI 192.168.206.138@tcp [ 1228.043371] Key type .llcrypt unregistered [ 1228.048904] Key type ._llcrypt unregistered [ 1254.403165] Key type ._llcrypt registered [ 1254.406872] Key type .llcrypt registered [ 1254.524856] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1255.771225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.837392] alg: No test for adler32 (adler32-zlib) [ 1256.983333] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 1257.259830] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1258.991165] Key type lgssc registered [ 1260.093902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1267.899799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1267.919560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.380276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.075276] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1279.973975] LustreError: 43270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.994523] LustreError: 43270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1282.674691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.114644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1287.133978] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1292.187804] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1296.129376] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1303.834230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1304.237867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.835944] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1314.493526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1316.891713] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1324.607493] Lustre: lustre-OST0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1325.024315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.033393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.043768] Lustre: Skipped 1 previous similar message [ 1329.597582] Lustre: server umount lustre-OST0000 complete [ 1349.600334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1349.606388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.634831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1349.642298] Lustre: Skipped 2 previous similar messages [ 1350.634592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.648202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1350.670408] Lustre: Skipped 1 previous similar message [ 1355.747791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.754841] Lustre: Skipped 1 previous similar message [ 1360.864900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.871972] Lustre: Skipped 1 previous similar message [ 1361.375342] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1361.659371] Lustre: server umount lustre-MDT0000 complete [ 1365.962383] LustreError: 43252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775195327 with bad export cookie 2404472168045365510 [ 1365.970620] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.973632] LustreError: 43252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1380.319929] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1380.546700] Lustre: server umount lustre-MDT0001 complete [ 1389.486348] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1392.014539] Key type lgssc unregistered [ 1392.339872] LNet: 46627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.355841] LNetError: 46627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.385298] LNet: Removed LNI 192.168.206.138@tcp [ 1394.169793] Key type .llcrypt unregistered [ 1394.173644] Key type ._llcrypt unregistered [ 1418.658929] Key type ._llcrypt registered [ 1418.662337] Key type .llcrypt registered [ 1418.799753] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1420.336346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.382891] alg: No test for adler32 (adler32-zlib) [ 1421.446961] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 1421.702412] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1423.440362] Key type lgssc registered [ 1424.544991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1431.626752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1431.652216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.267886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.961446] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1443.812380] LustreError: 48028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.828279] LustreError: 48028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1445.158513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.547858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1450.102588] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1455.773426] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1459.674440] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1466.543555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1466.942275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.237122] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1478.197319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1479.793178] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1486.200948] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1490.217865] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1502.631689] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-03 1h51m01s [ 1504.464961] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1507.299533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1507.316714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.342759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.922733] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1507.928862] Lustre: Skipped 2 previous similar messages [ 1509.000685] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1509.010496] Lustre: Skipped 3 previous similar messages [ 1512.834507] Lustre: server umount lustre-MDT0000 complete [ 1514.022724] LustreError: 48023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.737342] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1520.164520] LustreError: 48024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.196464] LustreError: 48024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1523.550167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.732623] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.975750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.048785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1524.976831] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1528.169240] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1531.458310] Lustre: DEBUG MARKER: Wait for df (36085) ... [ 1533.095779] Lustre: DEBUG MARKER: done [ 1537.507057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1537.519748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1537.530780] Lustre: Skipped 2 previous similar messages [ 1537.539128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1539.552902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1539.557525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.696064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1545.704884] Lustre: Skipped 3 previous similar messages [ 1551.327512] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1551.492879] Lustre: server umount lustre-OST0000 complete [ 1555.936620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.943178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1555.965843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1555.983493] Lustre: Skipped 4 previous similar messages [ 1561.414674] Lustre: server umount lustre-MDT0000 complete [ 1564.920108] LustreError: 50316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775195526 with bad export cookie 5521298537575063339 [ 1564.925535] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1564.936959] LustreError: 50316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1565.355551] Lustre: server umount lustre-MDT0001 complete [ 1576.912854] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1580.167933] Key type lgssc unregistered [ 1580.441664] LNet: 52588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.451238] LNetError: 52588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.470300] LNet: Removed LNI 192.168.206.138@tcp [ 1581.323168] Key type .llcrypt unregistered [ 1581.326292] Key type ._llcrypt unregistered [ 1620.075499] Key type ._llcrypt registered [ 1620.077565] Key type .llcrypt registered [ 1620.196756] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1621.386758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1621.428959] alg: No test for adler32 (adler32-zlib) [ 1622.452465] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 1622.695564] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1624.343189] Key type lgssc registered [ 1625.454916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1632.290834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1632.309552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.610144] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1633.634168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1633.754687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.234922] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1645.510587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.676446] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1645.731522] Lustre: 54780:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1646.099074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1650.248651] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1655.549456] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1659.324377] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1665.613797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1665.843684] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1666.101123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.031878] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1673.839144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1679.367970] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1687.616986] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-03 1h54m07s [ 1690.124497] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1697.670842] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1700.519298] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1700.593163] LustreError: 56780:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1720.915213] Lustre: lustre-MDT0000: Client 2794038d-b77b-4b14-8acc-cfec1e306f3b (at 192.168.206.38@tcp) reconnecting [ 1740.666947] LustreError: 56780:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1742.909644] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1745.302319] Lustre: DEBUG MARKER: done [ 1755.617940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1755.631759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.654672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.132126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1756.147141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.167417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.179024] Lustre: Skipped 1 previous similar message [ 1759.892383] Lustre: server umount lustre-OST0000 complete [ 1766.370141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.401491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1771.070953] Lustre: server umount lustre-MDT0000 complete [ 1775.656682] LustreError: 55514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775195737 with bad export cookie 7465936454668800416 [ 1775.668450] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1775.668806] LustreError: 55514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1776.208261] Lustre: server umount lustre-MDT0001 complete [ 1788.603692] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1791.851832] Key type lgssc unregistered [ 1792.299054] LNet: 58750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.323760] LNetError: 58750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.344223] LNet: Removed LNI 192.168.206.138@tcp [ 1793.349064] Key type .llcrypt unregistered [ 1793.354133] Key type ._llcrypt unregistered [ 1841.853323] Key type ._llcrypt registered [ 1841.857607] Key type .llcrypt registered [ 1842.006828] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1843.283077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1843.313423] alg: No test for adler32 (adler32-zlib) [ 1844.394833] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 1844.679172] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1846.384632] Key type lgssc registered [ 1847.615814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1851.160987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1851.178656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.567670] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1852.593908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1852.898621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.943225] Lustre: Failing over lustre-MDT0000 [ 1854.232852] Lustre: server umount lustre-MDT0000 complete [ 1861.706407] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:57:01 (1775195821) [ 1869.297818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.890051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.798460] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1884.837822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.981488] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1885.030520] Lustre: 61534:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1885.308402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1890.471940] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1896.603370] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1901.246983] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1909.118572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1909.401244] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1909.697399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.776433] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1919.719089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1923.249709] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1930.125170] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1940.457662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.469043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.478693] Lustre: Skipped 2 previous similar messages [ 1940.502346] Lustre: Skipped 2 previous similar messages [ 1944.549982] Lustre: server umount lustre-MDT0000 complete [ 1949.289237] LustreError: 61527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775195911 with bad export cookie 894053164547334761 [ 1949.291114] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1949.306886] LustreError: 61527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1949.593901] Lustre: server umount lustre-MDT0001 complete [ 1953.658739] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1958.708306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1967.073193] Lustre: 60462:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775195912/real 1775195912] req@00000000872b4092 x1861427717513344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775195928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1967.122419] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.621684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.207640] Lustre: 60462:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775195916/real 1775195916] req@0000000056f94895 x1861427717513472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775195932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1974.305535] LustreError: 60458:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009ec2c905 x1861427717514368/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1974.876705] LustreError: 63157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.894123] LustreError: 63157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1974.990677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.072275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1976.032197] Lustre: 60460:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775195921/real 1775195921] req@000000008e124027 x1861427717513856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775195937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1976.049142] LustreError: 63158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.072506] Lustre: 60460:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1978.082864] LustreError: 64966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.120608] LustreError: 64966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1980.311688] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1981.151251] Lustre: 60461:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775195926/real 1775195926] req@00000000e484a901 x1861427717513984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775195942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.463196] LustreError: 64967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.490105] LustreError: 64967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1988.578831] LustreError: 64968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.616975] LustreError: 64968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1991.591986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.973738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1996.073777] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1997.815320] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2004.923270] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2010.057242] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2017.764962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2017.773724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.803742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.858543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.864470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.884918] Lustre: Skipped 1 previous similar message [ 2021.902991] Lustre: Skipped 2 previous similar messages [ 2023.914566] Lustre: server umount lustre-MDT0000 complete [ 2026.979448] LustreError: 63158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.020180] LustreError: 63158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2029.379991] LustreError: 61528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775195991 with bad export cookie 894053164547337036 [ 2029.384073] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2029.388796] LustreError: 61528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2029.804590] Lustre: server umount lustre-MDT0001 complete [ 2037.529780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2047.456140] Lustre: 60461:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775195993/real 1775195993] req@00000000c567c98d x1861427717542272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775196009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2047.480244] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.552685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.688414] LustreError: 60458:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005a0bb9b5 x1861427717543296/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2055.080965] LustreError: 63157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.244120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.298550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2056.096140] Lustre: 60460:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775196001/real 1775196001] req@00000000bed6c154 x1861427717542784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775196017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2056.147217] Lustre: 60460:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2060.695718] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2071.658083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.462750] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2076.482440] Lustre: Skipped 1 previous similar message [ 2077.478220] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2085.111773] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2089.150924] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2097.126330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.131701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.158283] Lustre: Skipped 2 previous similar messages [ 2097.167258] Lustre: Skipped 1 previous similar message [ 2102.244332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.257165] Lustre: Skipped 3 previous similar messages [ 2102.835127] Lustre: server umount lustre-MDT0000 complete [ 2107.368114] LustreError: 63158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.398121] LustreError: 63158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2107.696285] LustreError: 62262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775196069 with bad export cookie 894053164547338415 [ 2107.706710] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2107.709116] LustreError: 62262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2108.069033] Lustre: server umount lustre-MDT0001 complete [ 2116.032860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2120.904724] Lustre: DEBUG MARKER: files compared the same [ 2134.011204] Lustre: server umount lustre-OST0000 complete [ 2154.317338] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2158.808509] Key type lgssc unregistered [ 2159.284921] LNet: 70480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2159.296329] LNetError: 70480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2159.330357] LNet: Removed LNI 192.168.206.138@tcp [ 2160.711305] Key type .llcrypt unregistered [ 2160.715613] Key type ._llcrypt unregistered [ 2190.538406] Key type ._llcrypt registered [ 2190.541176] Key type .llcrypt registered [ 2190.678896] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2191.981481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2192.126955] alg: No test for adler32 (adler32-zlib) [ 2193.521289] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 2193.980283] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2195.751236] Key type lgssc registered [ 2197.223498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2207.996435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2208.067119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.867806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.940161] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug +malloc all [ 2220.000198] LustreError: 71883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.032215] LustreError: 71883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2225.120890] LustreError: 71882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.087427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.514864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.337982] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug +malloc all [ 2240.951429] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2245.447806] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2253.909853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2254.422534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.433810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2262.718404] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug +malloc all [ 2269.943699] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2287.459959] Lustre: server umount lustre-OST0000 complete [ 2294.245059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2294.261638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.303306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2298.336545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2298.351887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.465944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2303.472209] Lustre: Skipped 2 previous similar messages [ 2303.496567] Lustre: 71193:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775196249/real 1775196249] req@00000000142f944c x1861428083456384/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1775196265 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2303.509888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.726889] Lustre: server umount lustre-MDT0000 complete [ 2308.582497] LustreError: 72594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.603944] LustreError: 72594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2309.600660] Lustre: 71192:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775196255/real 1775196255] req@00000000b356ec11 x1861428083457536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775196271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2309.639942] Lustre: 71192:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2310.078096] LustreError: 71861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775196271 with bad export cookie 8796796193687852285 [ 2310.082782] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2310.085824] LustreError: 71861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2310.500614] Lustre: server umount lustre-MDT0001 complete [ 2321.769692] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2325.079843] Key type lgssc unregistered [ 2325.401992] LNet: 75083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.413039] LNetError: 75083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.450280] LNet: Removed LNI 192.168.206.138@tcp [ 2326.733858] Key type .llcrypt unregistered [ 2326.736439] Key type ._llcrypt unregistered [ 2358.300697] Key type ._llcrypt registered [ 2358.302584] Key type .llcrypt registered [ 2358.460581] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2359.535709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2359.574818] alg: No test for adler32 (adler32-zlib) [ 2360.809112] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 2361.133730] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2362.855269] Key type lgssc registered [ 2364.814988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2374.698389] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2374.747398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2390.559834] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2395.679243] LustreError: 76456:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 2422.367857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.323600] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2436.959534] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2446.495165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.663368] Lustre: *** cfs_fail_loc=706, val=0*** [ 2447.271490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.364683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2452.456665] LustreError: 76481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.479942] LustreError: 76481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2452.497407] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2456.558270] LustreError: 76482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.567822] LustreError: 76482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2461.690910] LustreError: 77427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.711208] LustreError: 77427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2463.144102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.754994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2468.768723] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2475.807127] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2480.340286] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2483.176060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2483.196130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.226377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2484.196118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.220317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2484.227606] Lustre: Skipped 1 previous similar message [ 2488.560149] Lustre: server umount lustre-OST0000 complete [ 2494.432478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.433088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2494.462648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.475559] Lustre: Skipped 1 previous similar message [ 2499.553716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.572681] Lustre: Skipped 1 previous similar message [ 2499.922767] Lustre: server umount lustre-MDT0000 complete [ 2504.674540] LustreError: 78648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.700441] LustreError: 78648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2505.100905] LustreError: 76462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775196466 with bad export cookie 13867143215085595276 [ 2505.103716] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2505.127201] LustreError: 76462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2505.829504] Lustre: server umount lustre-MDT0001 complete [ 2519.897816] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2524.618315] Key type lgssc unregistered [ 2525.049832] LNet: 79722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2525.070398] LNetError: 79722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2525.101749] LNet: Removed LNI 192.168.206.138@tcp [ 2526.638949] Key type .llcrypt unregistered [ 2526.650168] Key type ._llcrypt unregistered [ 2543.667462] Key type ._llcrypt registered [ 2543.684693] Key type .llcrypt registered [ 2543.775743] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2544.836274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2544.866493] alg: No test for adler32 (adler32-zlib) [ 2545.907490] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 2546.123962] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2547.816062] Key type lgssc registered [ 2548.814103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2560.111565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2560.164531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.462172] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2585.927706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.442049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2592.763552] LustreError: 81772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.782562] LustreError: 81772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2593.707931] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2597.865560] LustreError: 81770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.892594] LustreError: 81770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2598.885875] LustreError: 81776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.948774] LustreError: 81776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2604.000287] LustreError: 81775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.063989] LustreError: 81775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2609.120737] LustreError: 81776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.194886] LustreError: 81776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2611.461810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2611.973601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.017028] LustreError: 81775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.064827] LustreError: 81775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2620.242119] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2632.009902] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2632.024291] Lustre: MGS: Client 063f4dd3-b448-444d-83a7-2e16abe0042c (at 0@lo) reconnecting [ 2632.045368] LustreError: 81786:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2632.414379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.525902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2637.672166] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2654.176238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2654.186149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.204710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.280228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.283951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.284025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.927322] Lustre: server umount lustre-OST0000 complete [ 2668.513392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.514081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2668.516135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.516144] Lustre: Skipped 1 previous similar message [ 2668.523934] Lustre: Skipped 1 previous similar message [ 2669.868588] Lustre: server umount lustre-MDT0000 complete [ 2673.636281] LustreError: 81776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.650569] LustreError: 81776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2674.368487] Lustre: server umount lustre-MDT0001 complete [ 2678.167360] Lustre: server umount MGS complete [ 2688.905979] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:10:48 (1775196648) [ 2717.594075] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 2727.122264] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2767.335795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.128502] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2782.494670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.787154] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2782.859821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2782.964843] Lustre: lustre-MDT0001: new disk, initializing [ 2783.033315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2783.069522] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2783.075857] Lustre: 88875:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2783.082480] LustreError: 88875:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2787.091689] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2798.357179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2798.656778] Lustre: lustre-OST0000: new disk, initializing [ 2798.663768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2798.781391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.423890] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2807.331087] Lustre: 89995:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000007466c70e x1861428452569088/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775196785 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2813.307609] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2821.083918] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2821.092829] Lustre: MGS: Client aae08ed4-f17a-48ce-b37b-9142fe378796 (at 0@lo) reconnecting [ 2821.112409] LustreError: 88861:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2821.162939] Lustre: 88158:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2821.221609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2821.361249] Lustre: lustre-MDT0000: new disk, initializing [ 2821.556920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.596386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2821.646607] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2822.694411] Lustre: 90582:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000539ca33e x1861428452575616/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1775196800 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2826.261878] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2827.825220] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2831.794945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2831.802821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2831.951301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2860.513068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2860.527344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.550240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.553095] Lustre: Skipped 1 previous similar message [ 2866.145721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2866.151388] Lustre: Skipped 2 previous similar messages [ 2873.311507] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2873.481841] Lustre: server umount lustre-OST0000 complete [ 2881.509545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.519924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2881.526488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2881.541836] Lustre: Skipped 2 previous similar messages [ 2881.571538] Lustre: Skipped 4 previous similar messages [ 2892.255250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2892.462816] Lustre: server umount lustre-MDT0000 complete [ 2896.877174] LustreError: 88847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.903863] LustreError: 88847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2903.675213] Lustre: server umount lustre-MDT0001 complete [ 2908.160951] Lustre: server umount MGS complete [ 2922.702755] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:14:42 (1775196882) [ 2943.919768] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2947.466506] Key type lgssc unregistered [ 2947.782562] LNet: 93053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2947.796013] LNetError: 93053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2947.816042] LNet: Removed LNI 192.168.206.138@tcp [ 2948.594979] Key type .llcrypt unregistered [ 2948.598124] Key type ._llcrypt unregistered [ 2971.320884] Key type ._llcrypt registered [ 2971.328868] Key type .llcrypt registered [ 2971.468839] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2972.848418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2972.865752] alg: No test for adler32 (adler32-zlib) [ 2973.883482] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 2974.227770] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2975.935495] Key type lgssc registered [ 2977.178253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2986.745853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2986.771812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.361305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.711702] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2998.755235] LustreError: 94260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.792102] LustreError: 94260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3003.893624] LustreError: 94261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.170968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.530349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3009.488222] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3016.417987] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3021.301394] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3029.847893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3030.312988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.421433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3038.347867] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3044.986425] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3060.194314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.198355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3060.207937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.211794] Lustre: Skipped 1 previous similar message [ 3065.320401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.333382] Lustre: Skipped 1 previous similar message [ 3065.560321] Lustre: server umount lustre-OST0000 complete [ 3070.432982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.448512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.453788] Lustre: Skipped 2 previous similar messages [ 3075.557775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.563671] Lustre: Skipped 1 previous similar message [ 3076.648810] Lustre: server umount lustre-MDT0000 complete [ 3080.673368] LustreError: 94261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.689684] LustreError: 94261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3081.776777] LustreError: 94242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775197043 with bad export cookie 18328716940344222946 [ 3081.813858] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3082.175031] Lustre: server umount lustre-MDT0001 complete [ 3098.348509] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3102.316202] Key type lgssc unregistered [ 3102.765439] LNet: 97461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3102.777849] LNetError: 97461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3102.795886] LNet: Removed LNI 192.168.206.138@tcp [ 3103.813955] Key type .llcrypt unregistered [ 3103.822782] Key type ._llcrypt unregistered [ 3122.540474] Key type ._llcrypt registered [ 3122.542596] Key type .llcrypt registered [ 3122.648849] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3123.463466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3123.620491] alg: No test for adler32 (adler32-zlib) [ 3124.712632] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 3125.003470] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3126.719224] Key type lgssc registered [ 3128.157421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3135.509508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3135.517800] LustreError: 98527:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 3135.525113] LustreError: 98530:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 3135.533147] LustreError: 98527:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3135.539360] LustreError: 98527:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3135.558823] LustreError: 98527:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3135.559505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.265091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3146.135098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.669033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3151.858933] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3158.590595] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing lsmod [ 3161.787622] LustreError: 99702:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 3161.794617] LustreError: 99701:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 3161.807066] LustreError: 99702:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3161.819349] LustreError: 99702:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3161.829227] LustreError: 99702:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3161.843105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3162.304051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.328363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3167.208158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3167.218970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.247145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3168.747481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3168.758915] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3168.783582] Lustre: Skipped 1 previous similar message [ 3171.827047] Lustre: server umount lustre-MDT0000 complete [ 3171.877903] LustreError: 98567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.916392] LustreError: 98567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3176.239760] LustreError: 98546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775197138 with bad export cookie 13645451491850422700 [ 3176.261423] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3176.929332] LustreError: 98564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.929935] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.978605] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3176.983609] Lustre: Skipped 2 previous similar messages [ 3178.079327] LustreError: 98999:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000010d3d89e x1861429059671808/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3178.245225] Lustre: server umount lustre-MDT0001 complete [ 3182.718282] Lustre: server umount lustre-OST0000 complete [ 3192.796497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.437421] LustreError: 100767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.581192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.489515] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3198.956186] LustreError: 100768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.064512] LustreError: 100767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.632142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.217546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3214.509612] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3222.552552] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3227.200251] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3235.521308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3235.817667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3238.913610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3243.146640] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3251.044293] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3259.365630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3259.375304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.395288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3263.461179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3263.463667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3263.485510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.573372] Lustre: server umount lustre-OST0000 complete [ 3270.624845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3273.696206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3273.696547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.705787] Lustre: Skipped 2 previous similar messages [ 3273.719697] Lustre: Skipped 1 previous similar message [ 3276.810499] Lustre: server umount lustre-MDT0000 complete [ 3278.817847] LustreError: 100768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.841019] LustreError: 100768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3281.865032] LustreError: 100748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775197243 with bad export cookie 13645451491850423631 [ 3281.870305] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.878670] LustreError: 100748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3282.540863] Lustre: server umount lustre-MDT0001 complete [ 3297.691599] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3300.837838] Key type lgssc unregistered [ 3301.191930] LNet: 103951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3301.204451] LNetError: 103951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3301.216228] LNet: Removed LNI 192.168.206.138@tcp [ 3302.183212] Key type .llcrypt unregistered [ 3302.185952] Key type ._llcrypt unregistered [ 3331.196420] Key type ._llcrypt registered [ 3331.199923] Key type .llcrypt registered [ 3331.312235] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3331.822507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3331.842569] alg: No test for adler32 (adler32-zlib) [ 3332.989550] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 3333.223761] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3334.943159] Key type lgssc registered [ 3335.862332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3344.539037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3344.556627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.241528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.474596] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3356.644964] LustreError: 105351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.697262] LustreError: 105351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3361.758772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.767868] LustreError: 105352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.287471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.608458] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3375.235379] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3381.174408] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3389.107266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.529523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.608241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3397.432678] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3404.553536] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3414.051816] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3417.635165] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3423.716196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.725342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3423.749660] Lustre: Skipped 1 previous similar message [ 3423.778304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3428.839114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.853450] Lustre: Skipped 2 previous similar messages [ 3429.497866] Lustre: server umount lustre-OST0000 complete [ 3439.078139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.087410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3439.088524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.110072] Lustre: Skipped 1 previous similar message [ 3440.743416] Lustre: server umount lustre-MDT0000 complete [ 3444.194849] LustreError: 105352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.227502] LustreError: 105352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3445.232738] LustreError: 105332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775197407 with bad export cookie 3497225387117177798 [ 3445.234501] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.245464] LustreError: 105332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3445.712293] Lustre: server umount lustre-MDT0001 complete [ 3459.490864] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3462.977158] Key type lgssc unregistered [ 3463.354906] LNet: 108640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3463.371343] LNetError: 108640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3463.404529] LNet: Removed LNI 192.168.206.138@tcp [ 3464.547904] Key type .llcrypt unregistered [ 3464.551191] Key type ._llcrypt unregistered [ 3484.846783] Key type ._llcrypt registered [ 3484.850524] Key type .llcrypt registered [ 3484.942772] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3486.070435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3486.323143] alg: No test for adler32 (adler32-zlib) [ 3487.502462] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 3487.811544] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3489.551507] Key type lgssc registered [ 3490.930892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3500.114659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3500.146446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.078866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.911760] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3511.295443] LustreError: 109844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3511.317835] LustreError: 109844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3516.386868] LustreError: 109845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.014623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.519341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3526.795896] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3533.807435] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3538.637780] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3546.955194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3547.111816] LustreError: 111441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3547.388627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.795497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3555.076927] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3562.732601] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3573.218089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.226751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3573.245911] Lustre: Skipped 1 previous similar message [ 3573.257710] Lustre: Skipped 1 previous similar message [ 3578.353944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.364465] Lustre: Skipped 1 previous similar message [ 3579.168850] Lustre: server umount lustre-OST0000 complete [ 3588.065284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3588.078816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.100952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3590.040230] Lustre: server umount lustre-MDT0000 complete [ 3593.697607] LustreError: 110955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.712371] LustreError: 110955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3594.804600] LustreError: 109826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775197556 with bad export cookie 8688446701464833145 [ 3594.811641] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3594.815821] LustreError: 109826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3595.225917] Lustre: server umount lustre-MDT0001 complete [ 3608.160279] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3611.144651] Key type lgssc unregistered [ 3611.400138] LNet: 113025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3611.405311] LNetError: 113025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3611.422132] LNet: Removed LNI 192.168.206.138@tcp [ 3612.157758] Key type .llcrypt unregistered [ 3612.159742] Key type ._llcrypt unregistered [ 3640.782221] Key type ._llcrypt registered [ 3640.787527] Key type .llcrypt registered [ 3640.883212] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3641.724358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3641.797709] alg: No test for adler32 (adler32-zlib) [ 3642.965969] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 3643.226926] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3644.911270] Key type lgssc registered [ 3646.551396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3657.063689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3657.123688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.887658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.169843] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3668.450952] LustreError: 114428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3668.467935] LustreError: 114428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3673.583758] LustreError: 114429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.299357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3675.750556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3680.133740] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3686.244970] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3690.684751] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3697.830793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3698.115026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.288919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3703.482301] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3708.970688] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3717.831917] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3722.690083] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3722.699846] Lustre: Skipped 2 previous similar messages [ 3725.479546] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3725.564618] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3729.151097] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3729.165895] Lustre: Skipped 2 previous similar messages [ 3734.554941] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3734.557581] Lustre: Skipped 1 previous similar message [ 3741.734491] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.38@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3741.749507] Lustre: Skipped 2 previous similar messages [ 3741.869669] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.206.38@tcp 192.168.0.[2,10]@tcp [ 3749.523957] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.38@tcp 192.168.0.[2,10]@tcp [ 3750.503087] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.38/24@tcp in log lustre-MDT0000 [ 3750.508989] Lustre: Skipped 2 previous similar messages [ 3756.113858] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.38/24@tcp [ 3764.162112] Lustre: server umount lustre-OST0000 complete [ 3769.833414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.847044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3769.854432] Lustre: Skipped 1 previous similar message [ 3769.865221] Lustre: Skipped 1 previous similar message [ 3774.945722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.952619] Lustre: Skipped 1 previous similar message [ 3778.099261] Lustre: server umount lustre-MDT0000 complete [ 3780.064341] LustreError: 115543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3780.097250] LustreError: 115543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3781.087455] Lustre: 113740:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775197726/real 1775197726] req@0000000036e91f2a x1861429603927680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775197742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3781.117253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.733930] LustreError: 116051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775197743 with bad export cookie 18268064884700237644 [ 3781.742926] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3782.182266] Lustre: server umount lustre-MDT0001 complete [ 3791.857424] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3794.261943] Key type lgssc unregistered [ 3794.581638] LNet: 118931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3794.590386] LNetError: 118931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3794.609559] LNet: Removed LNI 192.168.206.138@tcp [ 3795.436721] Key type .llcrypt unregistered [ 3795.439384] Key type ._llcrypt unregistered [ 3823.477725] Key type ._llcrypt registered [ 3823.481361] Key type .llcrypt registered [ 3823.572858] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3824.266237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3824.361075] alg: No test for adler32 (adler32-zlib) [ 3825.341597] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 3825.565819] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3827.207139] Key type lgssc registered [ 3828.001412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3836.031269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3836.054682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.940282] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3837.969955] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.38/24@tcp [ 3837.995180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.392928] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3848.160452] LustreError: 120482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.191660] LustreError: 120482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3853.287324] LustreError: 120483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.871625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.389342] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3854.422169] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.38/24@tcp [ 3854.523178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.574777] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3866.461230] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3871.022153] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3877.799183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3878.250372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.381512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3884.738502] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3890.975133] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3903.973383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3903.979815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3903.983180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3904.001743] Lustre: Skipped 1 previous similar message [ 3904.016876] Lustre: Skipped 1 previous similar message [ 3909.047769] Lustre: server umount lustre-OST0000 complete [ 3914.208483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.209526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3914.229286] Lustre: Skipped 1 previous similar message [ 3914.247223] Lustre: Skipped 1 previous similar message [ 3919.079384] Lustre: server umount lustre-MDT0000 complete [ 3919.329508] LustreError: 120482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3919.344206] LustreError: 120482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3922.720331] LustreError: 120464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775197884 with bad export cookie 14926121552633221834 [ 3922.722943] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3922.736235] LustreError: 120464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3923.074420] Lustre: server umount lustre-MDT0001 complete [ 3932.958480] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3935.481291] Key type lgssc unregistered [ 3935.718209] LNet: 123715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3935.728890] LNetError: 123715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3936.807407] LNet: Removed LNI 192.168.206.138@tcp [ 3937.516193] Key type .llcrypt unregistered [ 3937.518386] Key type ._llcrypt unregistered [ 3961.307217] Key type ._llcrypt registered [ 3961.309233] Key type .llcrypt registered [ 3961.397673] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3976.044978] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3977.345104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3977.445399] alg: No test for adler32 (adler32-zlib) [ 3978.488919] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 3978.689251] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3980.375193] Key type lgssc registered [ 3981.556206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4016.886990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4016.937198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4018.228284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4018.277712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4018.375584] Lustre: lustre-MDT0000: new disk, initializing [ 4018.510504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4018.531103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4022.749404] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4033.162739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.264337] Lustre: 127082:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4033.309123] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4033.313728] Lustre: Skipped 1 previous similar message [ 4033.363740] Lustre: lustre-MDT0001: new disk, initializing [ 4033.439821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4033.475897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4033.483816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4037.690235] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4046.515437] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4050.699214] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4059.067300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4059.553394] Lustre: lustre-OST0000: new disk, initializing [ 4059.558236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4059.714755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4065.918828] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4068.967487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4068.984535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4069.133715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4073.314619] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4077.785138] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4078.018325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4081.873985] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4082.081100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4084.707901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4084.709873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.719874] LustreError: Skipped 1 previous similar message [ 4084.735291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.742795] Lustre: Skipped 1 previous similar message [ 4089.828108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4089.841680] Lustre: Skipped 1 previous similar message [ 4090.128272] Lustre: server umount lustre-OST0000 complete [ 4094.944167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4094.947691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.953435] Lustre: Skipped 1 previous similar message [ 4094.962701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4094.970532] Lustre: Skipped 1 previous similar message [ 4099.941945] Lustre: server umount lustre-MDT0000 complete [ 4103.638751] LustreError: 127073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775198065 with bad export cookie 4065433195778085921 [ 4103.645612] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4103.648658] LustreError: 127073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4104.016705] Lustre: server umount lustre-MDT0001 complete [ 4112.283968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.787310] LustreError: 130893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.805047] LustreError: 130893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4112.876337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4117.357564] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4117.986625] LustreError: 130894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.108656] LustreError: 130893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.193611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4125.598414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4129.125741] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4135.218975] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4139.188752] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4146.383324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4146.792024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4154.161802] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4162.453462] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4168.504064] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4168.882754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4173.811861] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4174.129550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4182.979803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.100836] Lustre: lustre-OST0001: new disk, initializing [ 4183.110934] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4183.260300] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4189.669536] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4190.375507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4190.388918] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4190.564049] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4202.106481] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4202.524811] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4209.308575] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4209.627755] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4213.905602] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ffb04f57000.ost_server_uuid 50 [ 4216.121577] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ffb04f57000.ost_server_uuid in IDLE state after 0 sec [ 4231.650191] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4231.658297] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.674129] Lustre: Skipped 1 previous similar message [ 4231.682585] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4231.688666] Lustre: Skipped 1 previous similar message [ 4232.997864] Lustre: server umount lustre-OST0001 complete [ 4236.770882] LustreError: 132489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.780206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.798214] LustreError: 132489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4236.825017] Lustre: Skipped 1 previous similar message [ 4236.840251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4236.848725] Lustre: Skipped 1 previous similar message [ 4241.891465] LustreError: 132487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.902193] LustreError: 132487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4242.126239] Lustre: server umount lustre-OST0000 complete [ 4247.011763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4247.019520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4247.028912] Lustre: Skipped 2 previous similar messages [ 4247.030983] Lustre: Skipped 2 previous similar messages [ 4260.319388] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4260.539467] Lustre: server umount lustre-MDT0000 complete [ 4262.368779] LustreError: 131604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.385178] LustreError: 131604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4263.741075] LustreError: 132510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775198225 with bad export cookie 4065433195778086803 [ 4263.746935] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4263.765121] LustreError: 132510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4264.162411] Lustre: server umount lustre-MDT0001 complete [ 4284.504855] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4287.491633] Key type lgssc unregistered [ 4287.933373] LNet: 136740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4287.941079] LNetError: 136740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4287.968542] LNet: Removed LNI 192.168.206.138@tcp [ 4289.297220] Key type .llcrypt unregistered [ 4289.306477] Key type ._llcrypt unregistered [ 4325.881961] Key type ._llcrypt registered [ 4325.886680] Key type .llcrypt registered [ 4326.016744] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 4342.404334] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4343.916306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4343.937411] alg: No test for adler32 (adler32-zlib) [ 4345.207202] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 4345.600636] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4347.336357] Key type lgssc registered [ 4348.636030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4383.521050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4383.548046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.824520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4384.863912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4385.004528] Lustre: lustre-MDT0000: new disk, initializing [ 4385.187794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4385.202491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4389.952623] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4399.870847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.975487] Lustre: 140863:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4400.026607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4400.030951] Lustre: Skipped 1 previous similar message [ 4400.086191] Lustre: lustre-MDT0001: new disk, initializing [ 4400.191871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4400.221504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4400.229924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4403.803925] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4409.978142] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4413.601315] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4420.022180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4420.311625] Lustre: lustre-OST0000: new disk, initializing [ 4420.315545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4420.417473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4425.712341] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4428.400484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4428.413892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4428.555665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4432.534695] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4437.491568] Lustre: Failing over lustre-OST0000 [ 4439.009302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4439.017205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.030713] LustreError: Skipped 1 previous similar message [ 4439.033859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4439.063567] Lustre: Skipped 1 previous similar message [ 4439.069956] Lustre: Skipped 1 previous similar message [ 4440.111465] Lustre: server umount lustre-OST0000 complete [ 4459.539939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4459.787923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4459.802956] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4465.675931] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4467.274508] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4469.513827] Lustre: Failing over lustre-MDT0000 [ 4471.770887] Lustre: server umount lustre-MDT0000 complete [ 4471.783435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4471.799190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.338601] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.350886] Lustre: Skipped 1 previous similar message [ 4477.476212] LustreError: 140868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4477.498912] LustreError: 140868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4478.436227] LustreError: 140869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.455439] LustreError: 140869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4482.606843] LustreError: 142610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.234682] LustreError: 140870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.251957] LustreError: 140870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4490.075111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.181787] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4490.304263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4490.343207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.361194] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4490.361467] Lustre: lustre-OST0000: Recovery over after 0:23, of 3 clients 3 recovered and 0 were evicted. [ 4490.398200] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4491.300521] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4493.921717] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4495.853262] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4495.863503] Lustre: Skipped 1 previous similar message [ 4495.885241] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4495.959146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4506.083829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.085886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4506.093161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4506.107234] Lustre: Skipped 1 previous similar message [ 4508.713411] Lustre: server umount lustre-OST0000 complete [ 4516.323574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.324930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4516.342025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.342035] Lustre: Skipped 1 previous similar message [ 4516.344548] Lustre: Skipped 1 previous similar message [ 4518.576797] Lustre: server umount lustre-MDT0000 complete [ 4521.440292] LustreError: 140870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.475699] LustreError: 140870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4522.699714] LustreError: 140853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775198484 with bad export cookie 17528848867215711764 [ 4522.704097] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4522.710329] LustreError: 140853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4523.045747] Lustre: server umount lustre-MDT0001 complete [ 4534.366276] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4537.647074] Key type lgssc unregistered [ 4537.969702] LNet: 146101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4537.980622] LNetError: 146101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4538.017123] LNet: Removed LNI 192.168.206.138@tcp [ 4538.935913] Key type .llcrypt unregistered [ 4538.938334] Key type ._llcrypt unregistered [ 4573.882942] Key type ._llcrypt registered [ 4573.884793] Key type .llcrypt registered [ 4573.957225] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4574.790122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4574.803655] alg: No test for adler32 (adler32-zlib) [ 4575.859389] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 4576.063803] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4577.712594] Key type lgssc registered [ 4578.618887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4585.533893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4585.548938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.032400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.496447] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4596.197101] LustreError: 147604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.210629] LustreError: 147604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4597.324766] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4601.233160] LustreError: 147632:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4601.244812] LustreError: 147632:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4601.248510] LustreError: 147632:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4601.606743] Lustre: server umount lustre-MDT0000 complete [ 4615.908540] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 4627.521542] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4665.706740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.951910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4666.000698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4666.094884] Lustre: lustre-MDT0000: new disk, initializing [ 4666.161612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.216242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4670.123441] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4680.447768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4680.593046] Lustre: 150791:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4680.675487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4680.681994] Lustre: Skipped 1 previous similar message [ 4680.808768] Lustre: lustre-MDT0001: new disk, initializing [ 4680.954101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4680.988178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4680.999482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4686.001145] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4694.832218] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4700.354719] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4707.850777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4708.107517] Lustre: lustre-OST0000: new disk, initializing [ 4708.112987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4708.237252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.520872] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4717.623695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4717.646473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4717.711947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4723.156814] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4727.116838] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4728.802627] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:44:48 (1775198688) [ 5019.104514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5019.110783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.126306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5020.129734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5020.137055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5020.148832] Lustre: Skipped 1 previous similar message [ 5020.166233] Lustre: Skipped 2 previous similar messages [ 5025.250814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5025.258883] Lustre: Skipped 2 previous similar messages [ 5025.465618] Lustre: server umount lustre-MDT0000 complete [ 5030.367994] LustreError: 150801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5030.394554] LustreError: 150801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5030.878784] LustreError: 152530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775198992 with bad export cookie 9957510465156765882 [ 5030.898770] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5030.907248] LustreError: 152530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5035.487992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.488911] LustreError: 150798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5035.504462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5037.268980] Lustre: server umount lustre-MDT0001 complete [ 5048.181800] Lustre: server umount lustre-OST0000 complete [ 5061.872988] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 5075.297862] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5121.945576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.290960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5122.360570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5122.499962] Lustre: lustre-MDT0000: new disk, initializing [ 5122.628924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.656769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5127.173152] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5140.327465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.502914] Lustre: 157063:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5140.653442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5140.660692] Lustre: Skipped 1 previous similar message [ 5140.816690] Lustre: lustre-MDT0001: new disk, initializing [ 5140.962340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5141.046717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5141.062981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5146.893924] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5156.711823] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5161.679823] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5169.923518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5170.192503] Lustre: lustre-OST0000: new disk, initializing [ 5170.196467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5170.285267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5176.946708] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5178.435414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5178.451576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5178.559036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5185.937977] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5191.605841] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5191.814027] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5196.816833] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5197.081172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5199.329487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5199.331172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.338051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5199.350898] Lustre: Skipped 1 previous similar message [ 5199.357660] Lustre: Skipped 1 previous similar message [ 5205.587968] Lustre: server umount lustre-OST0000 complete [ 5212.656238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5212.668767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5212.696202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5212.703476] Lustre: Skipped 2 previous similar messages [ 5216.488456] Lustre: server umount lustre-MDT0000 complete [ 5219.808616] LustreError: 157074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5219.842752] LustreError: 157074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5220.437644] LustreError: 157054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775199182 with bad export cookie 9957510465156924705 [ 5220.450536] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5220.460982] LustreError: 157054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5220.839527] Lustre: server umount lustre-MDT0001 complete [ 5230.464422] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:53:10 (1775199190) [ 5246.945492] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 5256.041593] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5293.159139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.431996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5293.488387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5293.573248] Lustre: lustre-MDT0000: new disk, initializing [ 5293.709514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.729919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5297.677438] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5307.992213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.206214] Lustre: 163380:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5308.295073] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5308.302164] Lustre: Skipped 2 previous similar messages [ 5308.502413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5313.394909] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5322.263516] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5326.585458] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5334.118847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5334.406905] Lustre: lustre-OST0000: new disk, initializing [ 5334.409178] Lustre: Skipped 1 previous similar message [ 5334.414354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5334.424423] Lustre: Skipped 2 previous similar messages [ 5340.220605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5340.243647] Lustre: Skipped 1 previous similar message [ 5340.249347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5340.382576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5341.246496] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5349.548615] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5361.127769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5361.131610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.135635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5361.140876] LustreError: Skipped 1 previous similar message [ 5361.168745] Lustre: Skipped 2 previous similar messages [ 5363.669738] Lustre: server umount lustre-OST0000 complete [ 5369.824825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5382.114825] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5382.289515] Lustre: server umount lustre-MDT0000 complete [ 5385.944604] LustreError: 163372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775199347 with bad export cookie 9957510465156925594 [ 5385.954483] LustreError: 163372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5385.954715] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5386.352175] Lustre: server umount lustre-MDT0001 complete [ 5396.434614] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:55:56 (1775199356) [ 5414.496311] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 5427.468561] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5466.769596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.000681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5467.031633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5467.126597] Lustre: lustre-MDT0000: new disk, initializing [ 5467.227225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5467.230032] Lustre: Skipped 2 previous similar messages [ 5467.249356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5471.657927] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5483.303344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.416835] Lustre: 169314:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5483.722136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5488.723407] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5497.447725] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5501.560497] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5510.901266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5518.434836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5518.586686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5518.834603] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5527.791768] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5539.297062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5539.303211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5539.308503] LustreError: Skipped 1 previous similar message [ 5539.310646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5539.310654] Lustre: Skipped 8 previous similar messages [ 5539.345245] Lustre: Skipped 3 previous similar messages [ 5543.291384] Lustre: server umount lustre-OST0000 complete [ 5554.539631] Lustre: server umount lustre-MDT0000 complete [ 5554.680791] LustreError: 169325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.705890] LustreError: 169325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5558.979225] LustreError: 169304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775199520 with bad export cookie 9957510465156926672 [ 5558.987808] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5558.998483] LustreError: 169304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5559.455601] Lustre: server umount lustre-MDT0001 complete [ 5573.140039] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5576.786647] Key type lgssc unregistered [ 5577.225751] LNet: 172791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5577.234691] LNetError: 172791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5578.286222] LNet: Removed LNI 192.168.206.138@tcp [ 5579.627328] Key type .llcrypt unregistered [ 5579.629664] Key type ._llcrypt unregistered [ 5613.522696] Key type ._llcrypt registered [ 5613.526101] Key type .llcrypt registered [ 5613.705606] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5616.484860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5616.578406] alg: No test for adler32 (adler32-zlib) [ 5618.108300] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 5618.512609] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5620.231477] Key type lgssc registered [ 5622.474349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5634.626592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5634.695628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.663076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.209352] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5646.819235] LustreError: 174193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.841411] LustreError: 174193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5651.921597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5651.937057] LustreError: 174194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.516366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5657.196115] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5663.740351] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5667.585611] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5674.572204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5674.958874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5682.202127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5682.328841] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5689.567053] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5712.869690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5712.870671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.883590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5712.926400] Lustre: Skipped 1 previous similar message [ 5715.943768] Lustre: server umount lustre-OST0000 complete [ 5723.106432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5723.134384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5723.152819] Lustre: Skipped 2 previous similar messages [ 5727.235459] Lustre: server umount lustre-MDT0000 complete [ 5728.224162] LustreError: 174193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5728.269103] LustreError: 174193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5731.958309] LustreError: 174173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775199693 with bad export cookie 1195498733968702109 [ 5731.967254] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5731.986320] LustreError: 174173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5732.731377] Lustre: server umount lustre-MDT0001 complete [ 5747.299520] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5751.095105] Key type lgssc unregistered [ 5751.547159] LNet: 177412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5751.559669] LNetError: 177412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5751.589838] LNet: Removed LNI 192.168.206.138@tcp [ 5752.827256] Key type .llcrypt unregistered [ 5752.829710] Key type ._llcrypt unregistered [ 5784.421544] Key type ._llcrypt registered [ 5784.423626] Key type .llcrypt registered [ 5784.588803] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5785.712324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5785.805146] alg: No test for adler32 (adler32-zlib) [ 5787.030673] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 5787.363614] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5789.048922] Key type lgssc registered [ 5790.262842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5799.132469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5799.162659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.811168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5806.214416] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5810.165725] LustreError: 178815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.191804] LustreError: 178815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5815.265854] LustreError: 178816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.813770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.380558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5822.179700] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5828.950864] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5833.750496] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5841.453910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5841.868473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.021480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5849.231132] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5855.928656] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5864.529808] Lustre: lustre-OST0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 5866.477053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5866.481652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5866.485481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5866.521956] Lustre: Skipped 1 previous similar message [ 5869.630175] Lustre: lustre-OST0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 5869.648722] Lustre: Skipped 1 previous similar message [ 5870.035670] Lustre: server umount lustre-OST0000 complete [ 5876.527542] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8ffb05532800.ost_server_uuid 50 [ 5878.283897] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5897.185928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.191418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.209393] Lustre: Skipped 1 previous similar message [ 5897.235794] Lustre: Skipped 1 previous similar message [ 5900.368464] Lustre: server umount lustre-MDT0000 complete [ 5902.306584] LustreError: 179722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.327575] LustreError: 179722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5905.416177] LustreError: 178796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775199867 with bad export cookie 5058082048329669464 [ 5905.420742] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5905.428387] LustreError: 178796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5905.939224] Lustre: server umount lustre-MDT0001 complete [ 5918.218169] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:04:37 (1775199877) [ 5927.587652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.226940] LustreError: 182298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.373306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5932.854480] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5933.537181] LustreError: 182299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.754945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.115561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5948.438320] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5954.800911] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5959.222791] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5967.445053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5967.881813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5974.004922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5974.609504] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5981.177793] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5988.415321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.652729] Lustre: lustre-OST0001: new disk, initializing [ 5988.657632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5988.803402] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5995.579697] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5997.664512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5997.683429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5997.741664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6004.356900] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6008.300221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6008.319625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6018.537572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6018.551609] Lustre: Skipped 6 previous similar messages [ 6022.111133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6022.237989] Lustre: server umount lustre-OST0000 complete [ 6023.654274] LustreError: 183894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6023.677416] LustreError: 183894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6029.477208] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6029.670715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6033.806934] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6034.140471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6041.151914] LustreError: 185254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.170087] LustreError: 185254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6049.260501] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6049.266288] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6049.285992] LustreError: Skipped 1 previous similar message [ 6049.293982] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6049.293990] Lustre: Skipped 1 previous similar message [ 6049.363536] Lustre: Skipped 2 previous similar messages [ 6051.318934] Lustre: server umount lustre-OST0001 complete [ 6059.491337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6059.524665] Lustre: Skipped 1 previous similar message [ 6064.578970] Lustre: server umount lustre-MDT0000 complete [ 6069.536455] LustreError: 182278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775200031 with bad export cookie 5058082048329670528 [ 6069.538402] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6069.550063] LustreError: 182278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6070.285465] Lustre: server umount lustre-MDT0001 complete [ 6102.686402] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:07:42 (1775200062) [ 6110.909633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.321705] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6111.357677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6111.661427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6115.890556] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6125.656742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.878212] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6125.927130] Lustre: 188106:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6131.316865] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6138.359366] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6143.229742] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6151.019950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6151.236156] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6157.494211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6158.292627] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6164.627963] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6172.824441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6172.936929] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6180.190111] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6186.377968] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6192.125555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.133881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6192.141039] Lustre: Skipped 1 previous similar message [ 6192.149928] Lustre: Skipped 3 previous similar messages [ 6195.673378] Lustre: server umount lustre-OST0000 complete [ 6197.218389] LustreError: 189727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.241629] LustreError: 189727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6212.575926] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6212.587295] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.010981] Lustre: server umount lustre-OST0001 complete [ 6236.127165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6236.448753] Lustre: server umount lustre-MDT0000 complete [ 6240.493186] LustreError: 188843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775200202 with bad export cookie 5058082048329671942 [ 6240.497344] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6240.506644] LustreError: 188843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6240.911691] Lustre: server umount lustre-MDT0001 complete [ 6267.894641] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:10:27 (1775200227) [ 6284.841735] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 6296.701308] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 6338.306867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6338.605490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6338.648761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6338.803572] Lustre: lustre-MDT0000: new disk, initializing [ 6338.955870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6338.975152] Lustre: Skipped 3 previous similar messages [ 6339.007484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6344.210087] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6355.916188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.027644] Lustre: 195717:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6356.095536] LustreError: 195728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.108932] LustreError: 195728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6356.115261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6356.127034] Lustre: Skipped 1 previous similar message [ 6356.201402] Lustre: lustre-MDT0001: new disk, initializing [ 6356.325571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6356.334272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6360.809745] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6368.616293] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6372.821772] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6379.604325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6379.964611] Lustre: lustre-OST0000: new disk, initializing [ 6379.969416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6386.872704] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6388.818427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6388.834920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6388.949152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6395.611983] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6402.681472] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6402.963828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6409.388080] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6409.746514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6414.823804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.826170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.830723] LustreError: Skipped 2 previous similar messages [ 6414.832087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.832092] Lustre: Skipped 12 previous similar messages [ 6414.857198] Lustre: Skipped 4 previous similar messages [ 6419.153306] Lustre: server umount lustre-OST0000 complete [ 6429.727387] Lustre: server umount lustre-MDT0000 complete [ 6434.073734] LustreError: 195710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775200395 with bad export cookie 5058082048329673503 [ 6434.075497] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6434.086148] LustreError: 195710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6434.700591] Lustre: server umount lustre-MDT0001 complete [ 6444.282060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6450.419647] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6460.345286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.241813] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6472.365329] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6477.341742] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6484.343126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6484.763303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6484.770316] Lustre: Skipped 4 previous similar messages [ 6491.343500] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6498.490725] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6504.618324] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6504.874509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6510.324627] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6510.578394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6515.687264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6515.694915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.710635] Lustre: Skipped 2 previous similar messages [ 6518.994398] Lustre: server umount lustre-OST0000 complete [ 6526.619370] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6531.313619] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6562.803049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6570.126981] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6577.408593] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6579.798986] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6581.670207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.707705] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.710186] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6581.737447] Lustre: Skipped 1 previous similar message [ 6586.859893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6586.874738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.893026] LustreError: Skipped 1 previous similar message [ 6586.913058] Lustre: Skipped 5 previous similar messages [ 6591.092645] Lustre: server umount lustre-OST0000 complete [ 6601.556339] Lustre: server umount lustre-MDT0000 complete [ 6602.208951] LustreError: 199528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6602.254605] LustreError: 199528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6606.669504] LustreError: 199508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775200568 with bad export cookie 5058082048329674385 [ 6606.681570] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.695325] LustreError: 199508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6607.178708] Lustre: server umount lustre-MDT0001 complete [ 6616.922512] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:16:16 (1775200576) [ 6624.076568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6629.290734] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6638.072957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.648311] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6648.587978] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6652.694413] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6660.534024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6667.945332] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6674.169029] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6680.246512] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6680.462843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6686.330609] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6686.633889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6694.462954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6694.694632] Lustre: lustre-OST0001: new disk, initializing [ 6694.698338] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6701.669717] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6702.129286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6702.150859] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6702.280818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6710.370133] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6715.500319] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6715.698156] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6720.584909] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6720.784976] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6723.050884] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6723.061147] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.089209] Lustre: Skipped 5 previous similar messages [ 6736.864117] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6736.930540] Lustre: server umount lustre-OST0001 complete [ 6742.753861] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6742.928761] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6747.781734] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6747.979644] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6777.443748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.703388] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6777.706307] Lustre: Skipped 5 previous similar messages [ 6782.952000] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6782.974425] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6782.988054] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6783.959269] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6790.205437] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6791.804164] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6798.304070] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6798.320477] LustreError: Skipped 2 previous similar messages [ 6800.125533] Lustre: server umount lustre-OST0001 complete [ 6823.669341] Lustre: server umount lustre-OST0000 complete [ 6842.335471] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6842.546344] Lustre: server umount lustre-MDT0000 complete [ 6846.386447] LustreError: 207984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775200808 with bad export cookie 5058082048329675351 [ 6846.392341] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6846.398159] LustreError: 207984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6852.910587] Lustre: server umount lustre-MDT0001 complete [ 6879.855762] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:20:39 (1775200839) [ 6887.401543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6887.804542] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6887.854473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6892.798669] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6901.630782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.769959] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6901.824773] Lustre: 212942:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6907.151335] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6914.369402] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6919.076161] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6927.436632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6927.799571] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6935.057040] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6942.190246] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6953.744579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6953.869357] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6961.352203] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6968.086753] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6970.377274] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6975.538846] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6975.824022] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6980.829653] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6981.062740] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6986.680355] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ffb107ba800.ost_server_uuid 50 [ 6988.716720] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ffb107ba800.ost_server_uuid in IDLE state after 0 sec [ 6989.978676] Lustre: Permanently deactivating lustre-OST0001 [ 6990.011567] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6994.407534] Lustre: Permanently reactivating lustre-OST0001 [ 6994.424210] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6994.427428] Lustre: Skipped 2 previous similar messages [ 7000.038218] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7000.045511] LustreError: Skipped 3 previous similar messages [ 7000.049735] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.058351] Lustre: Skipped 6 previous similar messages [ 7000.061780] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7000.064109] Lustre: Skipped 23 previous similar messages [ 7013.858296] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7013.951890] Lustre: server umount lustre-OST0001 complete [ 7015.912065] LustreError: 214566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.953205] LustreError: 214566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 7024.801782] Lustre: server umount lustre-OST0000 complete [ 7036.675714] Lustre: server umount lustre-MDT0000 complete [ 7041.315945] LustreError: 213669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775201003 with bad export cookie 5058082048329676639 [ 7041.329307] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7041.833519] Lustre: server umount lustre-MDT0001 complete [ 7070.240993] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:23:49 (1775201029) [ 7078.797848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.970856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7078.984387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7078.992725] Lustre: Skipped 2 previous similar messages [ 7083.107599] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7093.054116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.258897] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7093.307555] Lustre: 219059:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7098.683297] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7105.492727] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7110.602393] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7118.143654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.330829] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7118.362227] Lustre: Permanently deactivating lustre-OST0000 [ 7118.388615] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7125.795660] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7127.655561] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7132.029498] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7139.471942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7139.601721] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7145.204431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7147.596594] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7154.678116] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7157.084249] Lustre: Permanently reactivating lustre-OST0000 [ 7157.101380] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7157.106337] Lustre: Skipped 2 previous similar messages [ 7165.092421] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7165.092926] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 000000009816f282, cur 1775201126 deadline 1775201119 last 1775201089 [ 7165.106049] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7165.108194] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7165.108202] Lustre: Skipped 1 previous similar message [ 7165.222613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7176.160335] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.176068] LustreError: Skipped 2 previous similar messages [ 7180.665595] Lustre: server umount lustre-OST0001 complete [ 7191.444712] Lustre: server umount lustre-OST0000 complete [ 7201.306131] Lustre: server umount lustre-MDT0000 complete [ 7206.807476] LustreError: 219053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775201168 with bad export cookie 5058082048329678305 [ 7206.809108] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7206.829982] LustreError: 219053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7207.240647] Lustre: server umount lustre-MDT0001 complete [ 7220.947521] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7224.441852] Key type lgssc unregistered [ 7224.782079] LNet: 223474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7224.800477] LNetError: 223474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7224.822454] LNet: Removed LNI 192.168.206.138@tcp [ 7225.815477] Key type .llcrypt unregistered [ 7225.817234] Key type ._llcrypt unregistered [ 7251.500169] Key type ._llcrypt registered [ 7251.503510] Key type .llcrypt registered [ 7251.595642] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7252.565556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7252.670774] alg: No test for adler32 (adler32-zlib) [ 7253.796084] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 7254.067804] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7255.719316] Key type lgssc registered [ 7257.000814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7268.171831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7268.186355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7269.771976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7275.136195] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7280.106733] LustreError: 224929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.148875] LustreError: 224929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7285.224697] LustreError: 224930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7285.871376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7286.095961] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7286.128507] Lustre: 224918:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7286.140737] Lustre: Permanently deactivating lustre-MDT0001 [ 7286.158545] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7286.557746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7291.788138] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7292.462914] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7298.669865] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7303.043987] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7311.076545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7311.396667] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7311.399631] LustreError: 226557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.569478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.614078] LustreError: 226546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7314.647619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7314.659668] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7318.645709] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7320.033863] LustreError: 226546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7320.044598] LustreError: 226546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7325.136045] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7330.282141] LustreError: 226545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7330.312829] LustreError: 226545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7332.855349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.242495] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7338.487160] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7338.502954] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7339.795897] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7346.434838] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7349.709720] Lustre: Permanently reactivating lustre-MDT0001 [ 7349.726874] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7349.735677] Lustre: Skipped 2 previous similar messages [ 7352.355313] Lustre: lustre-MDT0001: Client 537fa851-bbae-48dc-b8bc-f14580d7a6be (at 192.168.206.38@tcp) reconnecting [ 7357.629301] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7359.097836] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.114971] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7359.122386] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7359.142713] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7359.410400] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7366.154220] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7366.424911] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7372.018349] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7372.258791] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7375.103519] Lustre: Permanently deactivating lustre-MDT0001 [ 7375.122712] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7375.127353] Lustre: Skipped 2 previous similar messages [ 7381.172810] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7394.272977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7394.283459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7394.293293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7396.877776] Lustre: server umount lustre-MDT0000 complete [ 7399.904795] LustreError: 228920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7399.920641] LustreError: 228920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7401.145299] LustreError: 225652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775201363 with bad export cookie 4931747538066317389 [ 7401.147432] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7401.160309] LustreError: 225652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7401.679716] Lustre: server umount lustre-MDT0001 complete [ 7407.724968] Lustre: server umount lustre-OST0000 complete [ 7421.045659] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:29:40 (1775201380) [ 7432.840245] Lustre: server umount lustre-OST0001 complete [ 7440.468671] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 7454.352628] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7497.534275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.991429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7498.009680] Lustre: Skipped 2 previous similar messages [ 7498.059906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7498.157911] Lustre: lustre-MDT0000: new disk, initializing [ 7498.404501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7498.425403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7502.709663] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7514.574081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.735818] Lustre: 232593:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7514.803151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7514.807206] Lustre: Skipped 1 previous similar message [ 7514.864991] Lustre: lustre-MDT0001: new disk, initializing [ 7514.954768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7514.990445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7515.003290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7518.825836] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7527.268379] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7532.100570] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7541.151117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.565619] Lustre: lustre-OST0000: new disk, initializing [ 7541.578861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7541.700770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7549.796593] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7551.583712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7551.592300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7551.771412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7557.631514] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7560.451661] LustreError: 232599:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7569.837835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7570.146507] Lustre: lustre-OST0001: new disk, initializing [ 7570.151635] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7570.471116] LustreError: 232599:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7570.502799] LustreError: 234894:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7576.900822] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7580.305599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7580.330281] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7580.398547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7580.543096] LustreError: 234894:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7585.569666] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7587.826675] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7590.889292] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.898273] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7590.900147] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7590.900154] Lustre: Skipped 3 previous similar messages [ 7590.903472] Lustre: Skipped 4 previous similar messages [ 7596.010016] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7596.017530] Lustre: Skipped 2 previous similar messages [ 7596.543189] Lustre: server umount lustre-OST0001 complete [ 7601.126939] LustreError: 234318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7601.146837] LustreError: 234318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7613.414695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7613.421440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7613.428899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7613.454527] Lustre: Skipped 1 previous similar message [ 7613.474038] Lustre: Skipped 1 previous similar message [ 7619.305215] Lustre: server umount lustre-OST0000 complete [ 7626.730913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7626.740165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7626.754116] Lustre: Skipped 1 previous similar message [ 7626.765874] Lustre: Skipped 2 previous similar messages [ 7629.605369] Lustre: server umount lustre-MDT0000 complete [ 7634.396040] LustreError: 232586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775201596 with bad export cookie 4931747538066320203 [ 7634.401166] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.414586] LustreError: 232586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7640.992098] Lustre: server umount lustre-MDT0001 complete [ 7652.863815] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7655.651243] Key type lgssc unregistered [ 7656.018778] LNet: 237242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7656.025511] LNetError: 237242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7656.057196] LNet: Removed LNI 192.168.206.138@tcp [ 7657.007211] Key type .llcrypt unregistered [ 7657.009646] Key type ._llcrypt unregistered [ 7698.208197] Key type ._llcrypt registered [ 7698.211412] Key type .llcrypt registered [ 7698.349989] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7699.728259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7699.766559] alg: No test for adler32 (adler32-zlib) [ 7701.015125] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 7701.354189] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7703.079478] Key type lgssc registered [ 7704.310768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7712.374536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7712.401763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.720390] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7713.750877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7713.983552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7718.333732] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7727.590588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7727.737359] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7727.762170] Lustre: 239433:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7728.044212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7731.878337] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7737.428301] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7741.991475] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7749.691963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7749.921026] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7750.175344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7756.376552] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7757.434735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7761.266702] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7813.602456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.623193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.653583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7814.115303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7818.931791] Lustre: server umount lustre-OST0000 complete [ 7823.352754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7837.104328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7837.562490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.591405] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7839.621522] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7844.401567] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7850.887562] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7855.037705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7855.067659] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7890.923726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7890.928944] LustreError: Skipped 1 previous similar message [ 7890.928954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.939233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7890.949840] Lustre: Skipped 2 previous similar messages [ 7890.971892] Lustre: Skipped 3 previous similar messages [ 7896.038279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7896.245058] Lustre: server umount lustre-OST0000 complete [ 7901.156210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7901.160039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7901.179432] Lustre: Skipped 1 previous similar message [ 7901.187252] Lustre: Skipped 1 previous similar message [ 7906.276212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7906.293276] Lustre: Skipped 1 previous similar message [ 7906.597516] Lustre: server umount lustre-MDT0000 complete [ 7911.140698] LustreError: 239423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775201872 with bad export cookie 10904390925436287624 [ 7911.152795] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7911.158727] LustreError: 239423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7911.669852] Lustre: server umount lustre-MDT0001 complete [ 7923.398850] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7926.611447] Key type lgssc unregistered [ 7927.047781] LNet: 244390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7927.058743] LNetError: 244390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7927.074617] LNet: Removed LNI 192.168.206.138@tcp [ 7928.092847] Key type .llcrypt unregistered [ 7928.095703] Key type ._llcrypt unregistered [ 7954.788457] Key type ._llcrypt registered [ 7954.803320] Key type .llcrypt registered [ 7954.934690] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7955.841332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7955.884073] alg: No test for adler32 (adler32-zlib) [ 7957.131840] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 7957.434507] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7959.127191] Key type lgssc registered [ 7960.283848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7968.756821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7968.780058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.270903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7974.359485] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7980.513553] LustreError: 245792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7980.535342] LustreError: 245792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7982.493053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.804285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7986.512261] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7992.214673] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7995.961691] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8002.144987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.615167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.608486] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8014.128289] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8015.937109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8041.448417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8041.450191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.452523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8041.491718] Lustre: Skipped 1 previous similar message [ 8046.568603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8046.573584] Lustre: Skipped 2 previous similar messages [ 8047.229211] Lustre: server umount lustre-OST0000 complete [ 8051.682539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8051.696194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8051.700027] Lustre: Skipped 1 previous similar message [ 8051.713215] Lustre: Skipped 1 previous similar message [ 8056.801723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.815016] Lustre: Skipped 1 previous similar message [ 8057.162568] Lustre: server umount lustre-MDT0000 complete [ 8061.110125] LustreError: 245774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775202022 with bad export cookie 7393983294630550250 [ 8061.111766] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8061.126305] LustreError: 245774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8061.397592] Lustre: server umount lustre-MDT0001 complete [ 8071.664338] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 8074.572168] Key type lgssc unregistered [ 8074.969335] LNet: 249743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8074.979747] LNetError: 249743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8075.010239] LNet: Removed LNI 192.168.206.138@tcp [ 8075.965417] Key type .llcrypt unregistered [ 8075.966769] Key type ._llcrypt unregistered [ 8095.104550] Key type ._llcrypt registered [ 8095.107855] Key type .llcrypt registered [ 8095.217722] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8096.157252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8096.192167] alg: No test for adler32 (adler32-zlib) [ 8097.269529] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 8097.544661] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 8099.247224] Key type lgssc registered [ 8100.632487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8108.991051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8109.015385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.550734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8115.087373] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8120.807902] LustreError: 250949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8120.831501] LustreError: 250949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8124.299612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.751084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8129.630704] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8136.766277] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8141.418728] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8148.384078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8148.816591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8154.777840] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8156.980063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8161.356623] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8175.585787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8175.596603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8175.620303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8177.633627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8177.634342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8179.884392] Lustre: server umount lustre-OST0000 complete [ 8186.337812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8186.346714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.363794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8186.374629] Lustre: Skipped 1 previous similar message [ 8190.545543] Lustre: server umount lustre-MDT0000 complete [ 8192.994713] LustreError: 250945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8193.018458] LustreError: 250945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8194.603616] LustreError: 250930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775202156 with bad export cookie 11967046306286522531 [ 8194.606655] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8194.621800] LustreError: 250930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8195.012723] Lustre: server umount lustre-MDT0001 complete [ 8206.610132] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 8209.374806] Key type lgssc unregistered [ 8209.712956] LNet: 254275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8209.722453] LNetError: 254275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8209.747263] LNet: Removed LNI 192.168.206.138@tcp [ 8210.587238] Key type .llcrypt unregistered [ 8210.589755] Key type ._llcrypt unregistered [ 8237.423460] Key type ._llcrypt registered [ 8237.425516] Key type .llcrypt registered [ 8237.538430] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8238.743994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8238.869697] alg: No test for adler32 (adler32-zlib) [ 8240.039553] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 8240.345991] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 8242.023572] Key type lgssc registered [ 8243.173456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8252.034655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8252.062887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.617161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.048948] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8264.165788] LustreError: 255676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8264.189032] LustreError: 255676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8267.551596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.836959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8273.261591] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8281.045332] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8286.361803] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8295.169191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8295.866807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.961655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8303.737784] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8310.184127] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8337.892852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8337.894530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8337.907298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8337.928699] Lustre: Skipped 1 previous similar message [ 8342.300636] Lustre: server umount lustre-OST0000 complete [ 8348.134991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8348.136514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8348.153464] Lustre: Skipped 1 previous similar message [ 8348.172303] Lustre: Skipped 2 previous similar messages [ 8352.690570] Lustre: server umount lustre-MDT0000 complete [ 8353.249769] LustreError: 255676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8353.273424] LustreError: 255676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8356.601271] LustreError: 255658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775202318 with bad export cookie 1786509499878547144 [ 8356.603225] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8356.614139] LustreError: 255658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8357.027730] Lustre: server umount lustre-MDT0001 complete [ 8369.177670] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 8371.782526] Key type lgssc unregistered [ 8372.093091] LNet: 259632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8372.102219] LNetError: 259632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8372.113986] LNet: Removed LNI 192.168.206.138@tcp [ 8372.999966] Key type .llcrypt unregistered [ 8373.004575] Key type ._llcrypt unregistered [ 8391.556501] Key type ._llcrypt registered [ 8391.560581] Key type .llcrypt registered [ 8391.711264] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8392.785152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8392.945144] alg: No test for adler32 (adler32-zlib) [ 8394.099467] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 8394.376418] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 8396.087237] Key type lgssc registered [ 8397.503449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8405.493209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8405.521377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.961052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8411.675747] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8416.229836] LustreError: 260852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.262863] LustreError: 260852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8421.345959] LustreError: 260854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8421.563334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.025768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.117815] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8433.540895] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8437.967628] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8445.768478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8446.084796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.188110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8452.673920] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8458.994672] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8473.057542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8473.069796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8473.089988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8477.164672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8477.173089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8477.180203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8478.954631] Lustre: server umount lustre-OST0000 complete [ 8487.393393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.404806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8487.406580] Lustre: Skipped 1 previous similar message [ 8487.430646] Lustre: Skipped 2 previous similar messages [ 8489.821830] Lustre: server umount lustre-MDT0000 complete [ 8492.522597] LustreError: 260843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.542902] LustreError: 260843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8494.211539] LustreError: 260817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775202456 with bad export cookie 11272924427547736037 [ 8494.229774] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8494.232136] LustreError: 260817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8494.523823] Lustre: server umount lustre-MDT0001 complete [ 8506.531854] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 8509.066386] Key type lgssc unregistered [ 8509.346148] LNet: 264200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8509.359735] LNetError: 264200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8509.384061] LNet: Removed LNI 192.168.206.138@tcp [ 8510.135721] Key type .llcrypt unregistered [ 8510.137679] Key type ._llcrypt unregistered [ 8523.419509] Key type ._llcrypt registered [ 8523.423677] Key type .llcrypt registered [ 8523.559955] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8543.583219] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 8559.434171] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8560.578476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8560.676593] alg: No test for adler32 (adler32-zlib) [ 8561.777124] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 8562.032458] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 8563.706178] Key type lgssc registered [ 8564.730467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8599.273595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8599.362406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.795272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8600.849850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8601.007914] Lustre: lustre-MDT0000: new disk, initializing [ 8601.127769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8601.154749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8606.138909] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8617.040023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.141178] Lustre: 267705:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8617.213899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8617.220748] Lustre: Skipped 1 previous similar message [ 8617.364208] Lustre: lustre-MDT0001: new disk, initializing [ 8617.523361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8617.577596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8617.592257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8622.530229] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8631.685638] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8637.415740] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8645.287531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8645.770661] Lustre: lustre-OST0000: new disk, initializing [ 8645.775820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8645.961754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8652.500128] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8653.917599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8653.934273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8654.070641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8660.233124] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8664.519471] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8664.753269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8669.257283] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8669.469906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8674.794899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8674.797542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8674.805179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8674.829489] Lustre: Skipped 1 previous similar message [ 8677.336232] Lustre: server umount lustre-OST0000 complete [ 8684.007268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8684.017126] LustreError: Skipped 1 previous similar message [ 8684.028581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8684.051166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8684.057915] Lustre: Skipped 1 previous similar message [ 8685.031708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.060869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8685.064748] Lustre: Skipped 1 previous similar message [ 8688.009707] Lustre: server umount lustre-MDT0000 complete [ 8690.147401] LustreError: 267713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8690.168159] LustreError: 267713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8691.706116] LustreError: 267698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775202653 with bad export cookie 7147102749941802722 [ 8691.708912] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8691.713573] LustreError: 267698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8692.058584] Lustre: server umount lustre-MDT0001 complete [ 8701.399366] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:51:01 (1775202661) [ 8707.938251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8708.451598] LustreError: 271716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8708.518416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8712.634782] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8713.697223] LustreError: 271717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8718.817329] LustreError: 271716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8721.264414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.561565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8725.765747] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8731.540712] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8735.327828] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8742.509178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8742.841723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8748.936660] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8754.577489] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8764.391707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.400603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8764.413550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8764.437086] Lustre: Skipped 1 previous similar message [ 8769.511347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8769.528277] Lustre: Skipped 1 previous similar message [ 8770.161278] Lustre: server umount lustre-OST0000 complete [ 8774.624101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.654296] Lustre: Skipped 2 previous similar messages [ 8779.745044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8779.756196] Lustre: Skipped 3 previous similar messages [ 8780.160533] Lustre: server umount lustre-MDT0000 complete [ 8783.949161] LustreError: 271696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775202745 with bad export cookie 7147102749941803604 [ 8783.955260] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8783.964329] LustreError: 271696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8784.368415] Lustre: server umount lustre-MDT0001 complete [ 8794.526358] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 8797.093464] Key type lgssc unregistered [ 8797.395746] LNet: 274898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8797.402259] LNetError: 274898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8797.424487] LNet: Removed LNI 192.168.206.138@tcp [ 8798.227113] Key type .llcrypt unregistered [ 8798.233979] Key type ._llcrypt unregistered [ 8824.129472] Key type ._llcrypt registered [ 8824.132607] Key type .llcrypt registered [ 8824.228720] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 8840.287357] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 8841.572613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8841.693907] alg: No test for adler32 (adler32-zlib) [ 8842.825570] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 8843.238646] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 8844.935257] Key type lgssc registered [ 8846.222380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8894.042719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8894.057497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8895.192035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8895.227702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8895.316299] Lustre: lustre-MDT0000: new disk, initializing [ 8895.453058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8895.471498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8898.704475] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8911.063397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8911.230352] Lustre: 278695:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8911.341716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8911.348456] Lustre: Skipped 1 previous similar message [ 8911.503584] Lustre: lustre-MDT0001: new disk, initializing [ 8911.654219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8911.694458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8911.706994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8916.012893] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8925.306795] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8929.806392] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8937.236319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8937.603666] Lustre: lustre-OST2710: new disk, initializing [ 8937.608231] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8937.709674] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8944.123487] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8945.235402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8945.247517] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8945.372627] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8951.480145] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8958.359682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8958.580427] Lustre: lustre-OST03e8: new disk, initializing [ 8958.597881] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8958.718561] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8964.946335] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 8967.752492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8967.758681] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8967.826798] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8970.942954] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8978.387763] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8978.656642] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8983.292258] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8983.545037] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8988.143971] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8988.398846] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8992.801555] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8993.058299] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8998.895721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.898027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8998.906073] Lustre: Skipped 1 previous similar message [ 8998.911589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8998.931796] Lustre: Skipped 3 previous similar messages [ 9000.549748] Lustre: server umount lustre-MDT0000 complete [ 9004.751723] LustreError: 278687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775202966 with bad export cookie 14432951862222769831 [ 9004.753928] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9004.764845] LustreError: 278687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9005.333198] Lustre: server umount lustre-MDT0001 complete [ 9020.669880] Lustre: server umount lustre-OST2710 complete [ 9024.494313] Lustre: 276495:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775202970/real 1775202970] req@00000000cc83404a x1861435056520832/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775202986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9024.527462] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9024.540165] Lustre: Skipped 2 previous similar messages [ 9024.835933] Lustre: server umount lustre-OST03e8 complete [ 9043.435101] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 9054.875815] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 9096.369289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.614712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9096.636811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9096.707819] Lustre: lustre-MDT0000: new disk, initializing [ 9096.788876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9096.807248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9100.737925] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9115.213184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.405411] Lustre: 286422:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9115.483725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9115.487415] Lustre: Skipped 1 previous similar message [ 9115.608607] Lustre: lustre-MDT0001: new disk, initializing [ 9115.713988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9115.811203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9115.834641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9121.538580] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9130.878604] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9136.060572] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9143.900303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9144.359907] Lustre: lustre-OST0000: new disk, initializing [ 9144.370195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9144.521878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9150.890660] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9152.053421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9152.071136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9152.178813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9158.868374] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9163.833735] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9164.081884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9169.875636] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9170.158160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9172.961343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9172.970980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.977039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9173.015541] Lustre: Skipped 1 previous similar message [ 9178.086179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9178.098822] Lustre: Skipped 2 previous similar messages [ 9178.640765] Lustre: server umount lustre-OST0000 complete [ 9183.201829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.208430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9183.235328] Lustre: Skipped 1 previous similar message [ 9183.261855] Lustre: Skipped 1 previous similar message [ 9188.326136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9188.844958] Lustre: server umount lustre-MDT0000 complete [ 9193.443761] LustreError: 286429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9193.467705] LustreError: 286429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9194.539026] LustreError: 286414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775203156 with bad export cookie 14432951862222771511 [ 9194.552181] LustreError: 286414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9194.557367] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9195.233631] Lustre: server umount lustre-MDT0001 complete [ 9207.595312] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:59:27 (1775203167) [ 9209.866971] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9212.595526] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:59:31 (1775203171) [ 9216.545386] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 9249.000540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.266403] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9249.309573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9249.562374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9254.893827] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9265.500517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.606982] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9265.673402] Lustre: 291862:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9270.788479] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9278.219603] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9283.108202] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9304.234689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9304.476245] Lustre: Denying initial registration attempt from nid 192.168.206.138@tcp, specified as failover [ 9304.486248] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9304.501977] LustreError: 293476:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -99 [ 9304.515622] LustreError: 293476:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 9304.521512] LustreError: 293476:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 9304.641388] Lustre: server umount lustre-OST0000 complete [ 9304.645050] LustreError: 293476:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9312.224731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9312.240465] LustreError: Skipped 1 previous similar message [ 9312.253658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.273102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9312.285177] Lustre: Skipped 1 previous similar message [ 9318.436399] Lustre: server umount lustre-MDT0000 complete [ 9319.910067] LustreError: 293373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9319.938032] LustreError: 293373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9322.539652] LustreError: 291854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775203284 with bad export cookie 14432951862222772393 [ 9322.543596] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9322.551602] LustreError: 291854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9323.114433] Lustre: server umount lustre-MDT0001 complete [ 9336.279991] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 9339.183216] Key type lgssc unregistered [ 9339.381431] LNet: 294588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9339.386580] LNetError: 294588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9339.407423] LNet: Removed LNI 192.168.206.138@tcp [ 9340.155262] Key type .llcrypt unregistered [ 9340.157881] Key type ._llcrypt unregistered [ 9355.263582] Key type ._llcrypt registered [ 9355.266650] Key type .llcrypt registered [ 9355.443093] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 9356.669945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9356.722537] alg: No test for adler32 (adler32-zlib) [ 9357.948160] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 9358.281788] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 9359.999438] Key type lgssc registered [ 9361.777442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9425.285736] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 9434.117496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9434.155194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9435.409336] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9435.433829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9435.595916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.099454] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9449.724131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.834756] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9449.882600] Lustre: 297326:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9450.114209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9454.811415] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9461.312639] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9466.322582] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9487.021135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9487.230668] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9487.435751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9494.117471] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9500.487438] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9503.203715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9503.209315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9503.231376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9503.719120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9503.728707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9503.748744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9508.846808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9509.144079] Lustre: server umount lustre-OST0000 complete [ 9513.953163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9513.969305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9513.973394] Lustre: Skipped 2 previous similar messages [ 9519.074521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9519.094721] Lustre: Skipped 1 previous similar message [ 9527.263284] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9527.470287] Lustre: server umount lustre-MDT0000 complete [ 9531.158415] LustreError: 297318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775203493 with bad export cookie 13034371067708449452 [ 9531.164269] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9531.169023] LustreError: 297318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9531.608636] Lustre: server umount lustre-MDT0001 complete [ 9543.395115] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 9546.431338] Key type lgssc unregistered [ 9546.761713] LNet: 300549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9546.766553] LNetError: 300549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9546.776662] LNet: Removed LNI 192.168.206.138@tcp [ 9547.633796] Key type .llcrypt unregistered [ 9547.636648] Key type ._llcrypt unregistered [ 9575.311352] Key type ._llcrypt registered [ 9575.317596] Key type .llcrypt registered [ 9575.434018] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 9576.271661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9576.402670] alg: No test for adler32 (adler32-zlib) [ 9577.524267] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 9577.770777] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 9579.441210] Key type lgssc registered [ 9580.402898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9588.900469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9588.915961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9590.528966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9596.125972] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9600.996356] LustreError: 301950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9601.020710] LustreError: 301950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9606.120180] LustreError: 301951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9606.598917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.137583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9612.426864] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9619.318467] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9623.261596] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9633.075377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9633.529472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.845833] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9645.325994] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9662.432886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9662.441453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9662.446483] Lustre: Skipped 2 previous similar messages [ 9662.453183] Lustre: Skipped 1 previous similar message [ 9663.556672] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 9663.562761] Lustre: Skipped 1 previous similar message [ 9667.557391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9667.565799] Lustre: Skipped 2 previous similar messages [ 9672.674655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9672.683267] Lustre: Skipped 3 previous similar messages [ 9674.207415] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9674.370656] Lustre: server umount lustre-MDT0000 complete [ 9677.794509] LustreError: 301946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9677.829037] LustreError: 301946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9677.860949] LustreError: 301931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775203639 with bad export cookie 17530125746926202151 [ 9677.865388] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9677.885137] LustreError: 301931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9678.907218] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 9678.909748] Lustre: Skipped 1 previous similar message [ 9682.912164] LustreError: 303540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9682.912425] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.933076] LustreError: 303540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9684.294463] Lustre: server umount lustre-MDT0001 complete [ 9690.761819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9699.546158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9703.328199] LustreError: 301260:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005319ffbb x1861435826153472/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9703.558264] LustreError: 303556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.589835] LustreError: 303556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9703.655371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9703.730670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9708.484551] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9712.804077] LustreError: 303557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9712.812199] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9712.823319] LustreError: 303557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9717.093803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9717.574542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9722.196401] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9722.673973] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9728.119532] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9729.822644] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9734.419969] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9736.661539] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9743.334644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.338441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9743.367444] Lustre: Skipped 1 previous similar message [ 9743.368126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9743.372926] Lustre: Skipped 3 previous similar messages [ 9748.504688] Lustre: server umount lustre-OST0000 complete [ 9753.568365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.569201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9758.692641] LustreError: 305233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9758.704892] Lustre: server umount lustre-MDT0000 complete [ 9758.721468] LustreError: 305233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9762.553125] LustreError: 301932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775203724 with bad export cookie 17530125746926209732 [ 9762.557541] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9762.562356] LustreError: 301932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9762.847712] Lustre: server umount lustre-MDT0001 complete [ 9772.207429] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 9774.978691] Key type lgssc unregistered [ 9775.253081] LNet: 307665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9775.264625] LNetError: 307665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9775.291442] LNet: Removed LNI 192.168.206.138@tcp [ 9776.165151] Key type .llcrypt unregistered [ 9776.166612] Key type ._llcrypt unregistered [ 9801.047438] Key type ._llcrypt registered [ 9801.052054] Key type .llcrypt registered [ 9801.166219] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 9802.295660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9802.357587] alg: No test for adler32 (adler32-zlib) [ 9803.439760] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [ 9803.751564] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 9805.407650] Key type lgssc registered [ 9806.815395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9815.545514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9815.562509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9817.186713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9822.138831] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9826.277200] LustreError: 309065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9826.292647] LustreError: 309065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9831.307228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.402141] LustreError: 309066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.742230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9836.592316] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9843.366720] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9848.326204] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9852.389374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9852.390144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9852.407791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9852.433134] Lustre: Skipped 1 previous similar message [ 9857.504933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9857.513702] Lustre: Skipped 2 previous similar messages [ 9857.830257] Lustre: server umount lustre-MDT0000 complete [ 9858.538044] LustreError: 309065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9858.558291] LustreError: 309065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9861.673387] LustreError: 309045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775203823 with bad export cookie 1056545530243304769 [ 9861.681172] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9861.682314] LustreError: 309045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9861.945770] Lustre: server umount lustre-MDT0001 complete [ 9877.140053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9877.654148] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9877.682282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9878.009583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9883.227957] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9894.509139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9894.926143] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9895.032625] Lustre: 311426:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9895.404828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9900.150625] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9907.423307] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9912.063540] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9923.705955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9924.092627] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9924.121065] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9924.128899] LustreError: 313087:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 9924.146270] LustreError: 313087:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 9924.160695] LustreError: 313087:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 9924.296392] Lustre: server umount lustre-OST0000 complete [ 9924.300317] LustreError: 313087:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9936.243976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9936.555022] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9936.843606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9943.213372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9943.977368] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9950.225651] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9957.893087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9957.980927] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9958.052723] Lustre: lustre-OST0001: new disk, initializing [ 9958.058717] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9958.154252] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9964.057500] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 9964.664510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9964.674639] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9964.778563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9970.914566] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9975.279560] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9975.294910] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.320607] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9975.334391] Lustre: Skipped 1 previous similar message [ 9980.403657] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9980.413443] Lustre: Skipped 1 previous similar message [ 9980.570277] Lustre: server umount lustre-OST0001 complete [ 9985.505111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9985.509627] LustreError: 313555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.509928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9985.516299] Lustre: Skipped 2 previous similar messages [ 9985.537804] LustreError: 313555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9990.642280] LustreError: 313857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9990.658069] LustreError: 313857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9991.765958] Lustre: server umount lustre-OST0000 complete [ 9995.746604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9995.747711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.772814] Lustre: Skipped 1 previous similar message [ 9995.778746] Lustre: Skipped 4 previous similar messages [10002.169125] Lustre: server umount lustre-MDT0000 complete [10005.986943] LustreError: 311440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10006.016675] LustreError: 311440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10007.157338] LustreError: 311419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775203969 with bad export cookie 1056545530243305322 [10007.160579] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10007.165424] LustreError: 311419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10007.594533] Lustre: server umount lustre-MDT0001 complete [10022.570742] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [10027.430418] Key type lgssc unregistered [10027.794686] LNet: 316342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10027.801296] LNetError: 316342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10027.833393] LNet: Removed LNI 192.168.206.138@tcp [10029.056948] Key type .llcrypt unregistered [10029.059673] Key type ._llcrypt unregistered [10110.323651] Key type ._llcrypt registered [10110.328980] Key type .llcrypt registered [10110.439626] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [10126.225646] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [10127.473785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10127.590255] alg: No test for adler32 (adler32-zlib) [10128.730814] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [10129.032252] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [10130.744055] Key type lgssc registered [10131.928764] Lustre: Echo OBD driver; http://www.lustre.org/ [10171.996041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10172.071171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.297590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10173.319746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10173.401344] Lustre: lustre-MDT0000: new disk, initializing [10173.526393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10173.575241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10177.983797] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10189.600943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10189.751293] Lustre: 321244:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10189.814606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10189.818917] Lustre: Skipped 1 previous similar message [10189.885810] Lustre: lustre-MDT0001: new disk, initializing [10189.988095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10190.032801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10190.051578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10194.604524] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10202.825904] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10206.927454] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10213.984343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10214.246748] Lustre: lustre-OST0000: new disk, initializing [10214.250387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10214.359674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10220.493491] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10224.217866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10224.231398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10224.354520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10228.253174] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10232.891865] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10233.124918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10237.222345] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10237.396184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10239.968615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10239.971669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10239.986170] LustreError: Skipped 1 previous similar message [10240.005240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10240.022013] Lustre: Skipped 1 previous similar message [10243.045063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10243.055499] Lustre: Skipped 1 previous similar message [10245.515506] Lustre: server umount lustre-OST0000 complete [10251.745044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10251.761139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10251.780570] Lustre: Skipped 1 previous similar message [10251.798312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10253.281470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10255.681808] Lustre: server umount lustre-MDT0000 complete [10259.264248] LustreError: 321235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775204221 with bad export cookie 7239773095610225080 [10259.271558] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10259.276190] LustreError: 321235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10259.566438] Lustre: server umount lustre-MDT0001 complete [10268.638805] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:17:08 (1775204228) [10276.570635] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:17:16 (1775204236) [10285.304743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10285.958952] LustreError: 325592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.991400] LustreError: 325592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10286.068749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10291.187924] LustreError: 325592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10291.204361] LustreError: 325592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10291.577688] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10295.283620] LustreError: 325593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.385389] LustreError: 325592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.263705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10301.766304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10306.635703] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10313.266462] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10317.891266] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10325.333528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10325.818456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10333.077182] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10339.478337] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10349.745567] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10352.109297] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10353.731157] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10355.388378] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10357.085663] Lustre: Failing over lustre-MDT0000 [10357.390024] Lustre: server umount lustre-MDT0000 complete [10358.239996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10358.245870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10358.263305] LustreError: 325592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10366.944332] LustreError: 325593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10366.975615] LustreError: 325593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10377.162058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10377.183273] Lustre: 319470:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775204323/real 1775204323] req@00000000c777351f x1861436403946240/t0(0) o400->MGC192.168.206.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775204339 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10377.219893] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10383.907346] LustreError: 327436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10383.932967] LustreError: 327436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [10387.456938] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6478d745c2471944 [10387.466528] Lustre: MGC192.168.206.138@tcp: Connection restored to 0@lo (at 0@lo) [10387.834448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10387.896306] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10389.051352] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10392.778541] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10393.086053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10393.159785] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10393.207450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10401.003268] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10402.929960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10407.267307] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10409.322448] Lustre: Failing over lustre-MDT0000 [10409.544860] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [10409.554439] Lustre: Skipped 2 previous similar messages [10409.709707] Lustre: server umount lustre-MDT0000 complete [10413.539234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10413.542080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10413.571734] Lustre: Skipped 3 previous similar messages [10418.659490] LustreError: 327436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10418.682521] LustreError: 327436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10428.827989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10428.980739] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10429.287909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10429.352835] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10429.990227] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10433.929130] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10434.534877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10434.572205] Lustre: Skipped 2 previous similar messages [10434.617144] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10434.674590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10442.650811] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10444.292580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10455.009070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10455.011330] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10455.022079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10455.028798] Lustre: Skipped 3 previous similar messages [10455.047201] Lustre: Skipped 2 previous similar messages [10457.465825] Lustre: server umount lustre-MDT0000 complete [10462.259692] LustreError: 327208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775204424 with bad export cookie 7239773095610227796 [10462.261601] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10462.277259] LustreError: 327208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10463.116315] Lustre: server umount lustre-MDT0001 complete [10477.380990] Lustre: server umount lustre-OST0000 complete [10492.846156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10493.626556] LustreError: 331381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.662720] LustreError: 331381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [10493.754723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10498.760221] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10508.548071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.519806] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10519.556861] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10523.639375] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10529.543499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10535.096540] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10540.030205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10540.762270] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10549.222888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10549.234189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10549.260520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10552.194973] Lustre: server umount lustre-OST0000 complete [10562.521686] Lustre: server umount lustre-MDT0000 complete [10565.756189] LustreError: 331361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775204527 with bad export cookie 7239773095610228545 [10565.760617] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10565.773351] LustreError: 331361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10565.976502] Lustre: server umount lustre-MDT0001 complete [10576.889060] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [10579.550733] Key type lgssc unregistered [10579.889720] LNet: 334558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10579.896579] LNetError: 334558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10579.916254] LNet: Removed LNI 192.168.206.138@tcp [10580.701966] Key type .llcrypt unregistered [10580.703769] Key type ._llcrypt unregistered [10607.553219] Key type ._llcrypt registered [10607.555423] Key type .llcrypt registered [10607.692108] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [10608.782192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10608.905870] alg: No test for adler32 (adler32-zlib) [10609.977943] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [10610.178525] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [10611.863214] Key type lgssc registered [10612.949280] Lustre: Echo OBD driver; http://www.lustre.org/ [10620.619385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10620.651216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10621.997036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10626.900794] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10632.168175] LustreError: 336056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10632.196167] LustreError: 336056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10635.920241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10636.524855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10641.993910] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10648.524358] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10652.823362] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10659.495500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10659.921088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10665.627780] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10668.152470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10671.245829] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10682.852648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10682.870265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10682.883425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.102850] Lustre: server umount lustre-MDT0000 complete [10683.873428] LustreError: 336057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.874206] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.888725] LustreError: 336057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10683.914477] Lustre: Skipped 1 previous similar message [10687.265492] LustreError: 336037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775204649 with bad export cookie 363413529669967556 [10687.278690] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10687.665575] Lustre: server umount lustre-MDT0001 complete [10701.415619] Lustre: server umount lustre-OST0000 complete [10714.101034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10714.652864] LustreError: 339431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10714.670115] LustreError: 339431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10714.728068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10720.116429] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10720.234548] LustreError: 339432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10724.321663] LustreError: 339431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10730.986424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10731.429326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10736.233371] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10742.573309] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10747.454563] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10754.195727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10754.596551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10759.734479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10760.946902] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10767.597466] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10780.132397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10780.136787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10780.159151] Lustre: Skipped 2 previous similar messages [10780.179750] Lustre: Skipped 2 previous similar messages [10785.251019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10785.258014] Lustre: Skipped 2 previous similar messages [10785.899909] Lustre: server umount lustre-MDT0000 complete [10789.039447] LustreError: 339413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775204750 with bad export cookie 363413529669968676 [10789.046507] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10789.053235] LustreError: 339413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10789.588671] Lustre: server umount lustre-MDT0001 complete [10803.813388] Lustre: server umount lustre-OST0000 complete [10823.473665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10824.088774] LustreError: 343008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10824.105506] LustreError: 343008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10824.142092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10828.935967] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10838.187881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10843.066380] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10848.427577] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10852.553310] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10860.509448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10861.051753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10861.068789] Lustre: Skipped 1 previous similar message [10868.217788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10868.881736] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [10875.341868] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10883.552582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10883.556275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10883.560779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10883.568352] Lustre: Skipped 1 previous similar message [10887.600562] Lustre: server umount lustre-OST0000 complete [10893.793836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10893.794329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10893.805239] Lustre: Skipped 1 previous similar message [10893.831246] Lustre: Skipped 2 previous similar messages [10897.523668] Lustre: server umount lustre-MDT0000 complete [10898.913634] LustreError: 343008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10898.931823] LustreError: 343008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10900.877822] LustreError: 342988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775204862 with bad export cookie 363413529669969698 [10900.880375] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10900.892235] LustreError: 342988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10901.213617] Lustre: server umount lustre-MDT0001 complete [10912.386819] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [10915.399778] Key type lgssc unregistered [10915.733572] LNet: 346187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10915.741153] LNetError: 346187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10915.769748] LNet: Removed LNI 192.168.206.138@tcp [10916.745755] Key type .llcrypt unregistered [10916.749682] Key type ._llcrypt unregistered [10946.221339] Key type ._llcrypt registered [10946.223281] Key type .llcrypt registered [10946.327310] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [10947.229681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10947.375157] alg: No test for adler32 (adler32-zlib) [10948.551437] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [10948.915464] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [10950.647285] Key type lgssc registered [10951.724083] Lustre: Echo OBD driver; http://www.lustre.org/ [10960.141221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10960.167233] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10960.173584] LustreError: 347612:0:(osd_handler.c:8532:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10960.187919] LustreError: 347612:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000-osd failed (-22) [10960.196605] LustreError: 347612:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10960.203257] LustreError: 347612:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10960.216222] LustreError: 347612:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10968.187840] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10968.195773] LustreError: 348010:0:(osd_handler.c:8532:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10968.212773] LustreError: 348010:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-22) [10968.219934] LustreError: 348010:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10968.226397] LustreError: 348010:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10968.242125] LustreError: 348010:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10988.403390] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [10991.822276] Key type lgssc unregistered [10992.096321] LNet: 348996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10992.103725] LNetError: 348996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10992.121037] LNet: Removed LNI 192.168.206.138@tcp [10993.121501] Key type .llcrypt unregistered [10993.125603] Key type ._llcrypt unregistered [11014.404546] Key type ._llcrypt registered [11014.407524] Key type .llcrypt registered [11014.530760] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [11031.282524] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [11032.570795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11032.610805] alg: No test for adler32 (adler32-zlib) [11033.750609] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [11034.005404] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [11035.743294] Key type lgssc registered [11037.489749] Lustre: Echo OBD driver; http://www.lustre.org/ [11079.198254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11079.248336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.517994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11080.538425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11080.581886] Lustre: lustre-MDT0000: new disk, initializing [11080.652930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11080.670243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11084.996793] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11097.023557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11097.130142] Lustre: 352120:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11097.224647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11097.231472] Lustre: Skipped 1 previous similar message [11097.353811] Lustre: lustre-MDT0001: new disk, initializing [11097.466817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11097.504116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11097.511811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11102.898408] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11111.547636] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11116.076626] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11123.078170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11123.294514] Lustre: lustre-OST0000: new disk, initializing [11123.299288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11123.380317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11129.816423] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11131.001639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11131.014787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11131.117115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11137.880565] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11142.623882] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11142.846752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11147.702400] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11147.996852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11151.850796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11151.858612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11151.862391] LustreError: Skipped 1 previous similar message [11151.884542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11156.573806] Lustre: server umount lustre-OST0000 complete [11162.081730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11162.083578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11162.102334] Lustre: Skipped 2 previous similar messages [11162.105678] Lustre: Skipped 1 previous similar message [11167.204571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11167.215281] Lustre: Skipped 2 previous similar messages [11171.300872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11171.314249] Lustre: Skipped 1 previous similar message [11175.391570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11175.741204] Lustre: server umount lustre-MDT0000 complete [11180.542758] LustreError: 352111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775205142 with bad export cookie 14123126123529161298 [11180.567871] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11181.144234] Lustre: server umount lustre-MDT0001 complete [11190.441389] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11192.684236] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:32:32 (1775205152) [11199.029826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.373550] LustreError: 356176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11199.383787] LustreError: 356176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11199.425131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11203.314677] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11204.579271] LustreError: 356177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11209.704553] LustreError: 356176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11211.527813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11211.924922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11216.214440] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11224.697218] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11229.915531] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11240.407457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11241.122410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11247.327892] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11254.622986] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11261.064284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11261.203602] Lustre: lustre-OST0001: new disk, initializing [11261.208598] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11261.332457] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11267.609741] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11269.213079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11269.223142] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11269.368043] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11275.640670] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11279.840277] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11279.849570] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11279.851607] LustreError: Skipped 1 previous similar message [11279.852238] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11279.865896] Lustre: Skipped 1 previous similar message [11285.551668] Lustre: server umount lustre-OST0001 complete [11287.590935] LustreError: 357919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11295.219546] LustreError: 357919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11295.241579] LustreError: 357919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11302.897555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11302.904446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11302.916121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11302.943417] Lustre: Skipped 4 previous similar messages [11305.447676] LustreError: 357772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.477653] LustreError: 357772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11308.867342] Lustre: server umount lustre-OST0000 complete [11314.145385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11314.161619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11314.178185] Lustre: Skipped 1 previous similar message [11319.352711] Lustre: server umount lustre-MDT0000 complete [11323.379633] LustreError: 356157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775205285 with bad export cookie 14123126123529162180 [11323.388409] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11323.393209] LustreError: 356157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11329.864152] Lustre: server umount lustre-MDT0001 complete [11340.831767] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [11343.549274] Key type lgssc unregistered [11343.936477] LNet: 360536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11343.949960] LNetError: 360536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11343.966360] LNet: Removed LNI 192.168.206.138@tcp [11344.869355] Key type .llcrypt unregistered [11344.872125] Key type ._llcrypt unregistered [11399.402788] Key type ._llcrypt registered [11399.405479] Key type .llcrypt registered [11399.501412] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [11400.695822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11400.740388] alg: No test for adler32 (adler32-zlib) [11401.781890] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [11401.971262] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [11403.615581] Key type lgssc registered [11404.819837] Lustre: Echo OBD driver; http://www.lustre.org/ [11412.716427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11412.734446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11414.020087] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11414.052933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11414.320154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11419.424643] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11428.236026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11428.346157] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11428.376722] Lustre: 363321:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11428.633156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11432.573536] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11438.231400] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11442.233399] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11444.195982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11444.200471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11444.220188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11447.264582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11447.274927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11452.348221] Lustre: server umount lustre-MDT0000 complete [11456.273752] LustreError: 363312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775205418 with bad export cookie 7715013636833219150 [11456.275644] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11456.284528] LustreError: 363312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11456.591437] Lustre: server umount lustre-MDT0001 complete [11466.645657] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:37:06 (1775205426) [11473.228522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11473.727197] LustreError: 365606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11473.742484] LustreError: 365606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11473.793553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11479.013016] LustreError: 365607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11479.125892] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11484.153408] LustreError: 365606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11489.252258] LustreError: 365607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11490.012801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11490.444794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11495.560754] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11502.440864] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11507.411199] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11514.478636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11514.699934] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11514.931108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11521.513670] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11527.655913] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11536.157753] Lustre: Permanently deactivating lustre-OST0000 [11536.177521] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11544.111897] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11545.755449] LustreError: 368255:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11545.762640] LustreError: 368255:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11545.773618] LustreError: 368255:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11549.424809] LustreError: 368304:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11549.429193] LustreError: 368304:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11557.903957] Lustre: server umount lustre-OST0000 complete [11561.023270] LustreError: 368554:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11561.038945] LustreError: 368554:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11564.011678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11564.023068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11564.042717] Lustre: Skipped 1 previous similar message [11564.068477] Lustre: Skipped 2 previous similar messages [11569.124838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11569.131286] Lustre: Skipped 1 previous similar message [11573.393204] Lustre: server umount lustre-MDT0000 complete [11574.252338] LustreError: 366315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11574.272270] LustreError: 366315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11578.263683] LustreError: 365586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775205540 with bad export cookie 7715013636833219731 [11578.265831] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11578.291963] LustreError: 365586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11588.929291] Lustre: server umount lustre-MDT0001 complete [11599.269868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.147805] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11607.222237] LustreError: 369698:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11614.590667] Lustre: 370037:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.138@tcp [11617.217231] Lustre: server umount MGS complete [11630.564682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11631.199131] LustreError: 370762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11631.231666] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11631.237208] Lustre: Skipped 1 previous similar message [11631.321029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11636.624481] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11645.833037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.131104] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11650.640712] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11656.281424] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11660.361200] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11662.253874] Lustre: Permanently reactivating lustre-OST0000 [11662.278572] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11662.284703] Lustre: Skipped 2 previous similar messages [11690.463143] LustreError: 372314:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11698.532058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11699.001670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11699.009506] Lustre: Skipped 1 previous similar message [11704.848974] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11710.473360] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11716.576350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11716.583139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11716.586152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11716.592749] Lustre: Skipped 1 previous similar message [11716.604458] Lustre: Skipped 1 previous similar message [11722.222827] Lustre: server umount lustre-OST0000 complete [11726.816570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11726.823133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11726.831737] Lustre: Skipped 1 previous similar message [11726.844039] Lustre: Skipped 3 previous similar messages [11732.391181] Lustre: server umount lustre-MDT0000 complete [11736.263555] LustreError: 370742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775205698 with bad export cookie 7715013636833220781 [11736.284477] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11736.621239] Lustre: server umount lustre-MDT0001 complete [11749.080848] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [11752.420786] Key type lgssc unregistered [11752.776550] LNet: 374771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11752.790277] LNetError: 374771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11752.825300] LNet: Removed LNI 192.168.206.138@tcp [11753.632774] Key type .llcrypt unregistered [11753.636584] Key type ._llcrypt unregistered [11774.437459] Key type ._llcrypt registered [11774.439430] Key type .llcrypt registered [11774.570248] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [11792.980168] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [11793.885569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11794.112691] alg: No test for adler32 (adler32-zlib) [11795.243463] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [11795.499707] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [11797.151833] Key type lgssc registered [11798.300820] Lustre: Echo OBD driver; http://www.lustre.org/ [11834.749134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11834.782831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11836.062364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11836.092516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11836.210571] Lustre: lustre-MDT0000: new disk, initializing [11836.404680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11836.444650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11840.862025] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11850.792701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11850.875925] Lustre: 377897:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11850.916269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11850.919874] Lustre: Skipped 1 previous similar message [11850.975233] Lustre: lustre-MDT0001: new disk, initializing [11851.057027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11851.093057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11851.106716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11854.819845] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11862.166872] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11866.319768] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11873.248923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11873.489735] Lustre: lustre-OST0000: new disk, initializing [11873.492567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11873.578850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11879.987075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11880.004338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11880.047283] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11880.156362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11888.702466] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11893.834755] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11894.088601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11899.275628] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11899.535870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11906.017700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11906.028673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11906.034580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11907.794624] Lustre: server umount lustre-OST0000 complete [11916.257410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11916.285113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11917.276522] Lustre: Skipped 2 previous similar messages [11917.292766] Lustre: Skipped 2 previous similar messages [11921.377296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11921.385794] Lustre: Skipped 1 previous similar message [11925.473291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11925.492255] Lustre: Skipped 1 previous similar message [11927.525559] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11927.942202] Lustre: server umount lustre-MDT0000 complete [11933.319199] LustreError: 377888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775205895 with bad export cookie 17095462352697319775 [11933.321047] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11933.335575] LustreError: 377888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11933.869611] Lustre: server umount lustre-MDT0001 complete [11945.580956] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11947.937622] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:45:07 (1775205907) [11955.276111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11955.837939] LustreError: 381953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.870047] LustreError: 381953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11955.927612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11960.837358] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11961.322124] LustreError: 381954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11965.475072] LustreError: 381953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11969.646075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11970.112459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11973.643668] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [11979.913955] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11984.060267] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11990.753871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11991.109538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11997.825218] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12003.744421] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12006.108177] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12008.914725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12014.570163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12014.581741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12014.604735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12015.590984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12020.673400] Lustre: server umount lustre-OST0000 complete [12025.824810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12025.826726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12025.843039] Lustre: Skipped 2 previous similar messages [12025.867046] Lustre: Skipped 3 previous similar messages [12031.312149] Lustre: server umount lustre-MDT0000 complete [12035.049644] LustreError: 381934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775205996 with bad export cookie 17095462352697320650 [12035.058706] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12035.302786] Lustre: server umount lustre-MDT0001 complete [12047.216533] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12050.690106] Key type lgssc unregistered [12050.999700] LNet: 385326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12051.005392] LNetError: 385326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12051.021471] LNet: Removed LNI 192.168.206.138@tcp [12051.724647] Key type .llcrypt unregistered [12051.728311] Key type ._llcrypt unregistered [12077.311568] Key type ._llcrypt registered [12077.316014] Key type .llcrypt registered [12077.459672] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12077.890454] Key type .llcrypt unregistered [12077.901814] Key type ._llcrypt unregistered [12096.584433] Key type ._llcrypt registered [12096.589176] Key type .llcrypt registered [12096.735559] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12097.805938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12097.886785] alg: No test for adler32 (adler32-zlib) [12098.972679] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [12099.114164] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12100.745244] Key type lgssc registered [12102.617852] Lustre: Echo OBD driver; http://www.lustre.org/ [12111.565620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12111.599195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12113.180062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12118.032511] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12122.593706] LustreError: 387507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.609331] LustreError: 387507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12126.859592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12127.171373] LustreError: 387508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.191798] LustreError: 387508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12127.204137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12132.325638] LustreError: 387502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12132.338118] LustreError: 387502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12133.658435] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12137.462865] LustreError: 387507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12137.479348] LustreError: 387507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12137.485287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12139.713895] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12142.569220] LustreError: 387508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12142.592547] LustreError: 387508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12146.701405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.016314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12151.743150] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12162.539212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12162.543784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12162.552617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12162.560094] Lustre: Skipped 1 previous similar message [12162.577738] Lustre: Skipped 1 previous similar message [12166.857924] Lustre: server umount lustre-OST0000 complete [12172.770401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12172.779255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12172.787898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12172.798612] Lustre: Skipped 1 previous similar message [12172.823025] Lustre: Skipped 1 previous similar message [12174.707246] Lustre: server umount lustre-MDT0000 complete [12177.888506] LustreError: 389470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12177.922119] LustreError: 389470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12178.804338] LustreError: 387487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775206140 with bad export cookie 4040792652147174896 [12178.810477] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12178.814959] LustreError: 387487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12179.428320] Lustre: server umount lustre-MDT0001 complete [12193.290160] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12196.691594] Key type lgssc unregistered [12197.072787] LNet: 390495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12197.090621] LNetError: 390495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12197.110859] LNet: Removed LNI 192.168.206.138@tcp [12198.020377] Key type .llcrypt unregistered [12198.025933] Key type ._llcrypt unregistered [12226.801762] Key type ._llcrypt registered [12226.807674] Key type .llcrypt registered [12226.896990] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12228.114560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12228.380457] alg: No test for adler32 (adler32-zlib) [12229.367827] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [12229.544420] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12231.240733] Key type lgssc registered [12232.669138] Lustre: Echo OBD driver; http://www.lustre.org/ [12240.741075] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12240.761884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12256.287364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12261.408208] LustreError: 391867:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [12288.031677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12294.252239] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12300.888314] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12307.958711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12308.680934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12308.776680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12313.226686] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12314.087723] LustreError: 391892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12314.118617] LustreError: 391892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12319.204774] LustreError: 391893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12322.289788] LustreError: 392790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12322.321915] LustreError: 392790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12323.000287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12323.510378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12328.607605] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12336.053493] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12341.380509] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12348.901482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12348.912283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12348.927775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12349.431149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12349.449365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12349.457270] Lustre: Skipped 1 previous similar message [12350.219922] Lustre: server umount lustre-OST0000 complete [12356.076154] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12356.083896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.107836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12357.157365] Lustre: server umount lustre-MDT0000 complete [12359.648158] LustreError: 392794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12359.652781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12361.159404] LustreError: 391875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775206323 with bad export cookie 17572738347916671961 [12361.163692] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12361.169386] LustreError: 391875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12361.499153] Lustre: server umount lustre-MDT0001 complete [12373.641286] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12377.853848] Key type lgssc unregistered [12378.376942] LNet: 395083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12378.392731] LNetError: 395083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12378.438691] LNet: Removed LNI 192.168.206.138@tcp [12379.775232] Key type .llcrypt unregistered [12379.778580] Key type ._llcrypt unregistered [12411.699593] Key type ._llcrypt registered [12411.702709] Key type .llcrypt registered [12411.856478] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12412.904396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12412.972289] alg: No test for adler32 (adler32-zlib) [12414.010555] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [12414.279473] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12415.959231] Key type lgssc registered [12416.984811] Lustre: Echo OBD driver; http://www.lustre.org/ [12424.518976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12424.539344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.074282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12430.459932] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12435.433220] LustreError: 396485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12435.458323] LustreError: 396485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12439.430233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12439.938610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12444.870976] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12450.855643] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12455.584287] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12462.979152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12463.220040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12468.878964] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12470.392789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12475.471051] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12480.995264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12481.007660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12481.007805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12481.028091] Lustre: Skipped 2 previous similar messages [12481.057473] Lustre: Skipped 2 previous similar messages [12483.646606] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [12486.126131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12486.142657] Lustre: Skipped 2 previous similar messages [12486.722620] Lustre: server umount lustre-MDT0000 complete [12488.750741] LustreError: 396482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12489.444562] LustreError: 398103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.38@tcp arrived at 1775206451 with bad export cookie 14180962207519916416 [12489.473540] LustreError: 398103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12491.237228] LustreError: 396480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12491.259983] LustreError: 396480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12491.719029] LustreError: 396465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775206453 with bad export cookie 14180962207519915520 [12491.735452] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12491.736688] LustreError: 396465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12495.328221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12495.339047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12495.365025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12496.353393] LustreError: 397603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12496.378985] LustreError: 397603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12498.138175] Lustre: server umount lustre-OST0000 complete [12501.499014] LustreError: 396486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12501.538961] LustreError: 396486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12506.275847] Lustre: server umount lustre-MDT0001 complete [12518.763090] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12521.731264] Key type lgssc unregistered [12522.071129] LNet: 399818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12522.096286] LNetError: 399818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12522.128349] LNet: Removed LNI 192.168.206.138@tcp [12523.001138] Key type .llcrypt unregistered [12523.003161] Key type ._llcrypt unregistered [12549.009290] Key type ._llcrypt registered [12549.014568] Key type .llcrypt registered [12549.141441] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12550.265739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12550.411073] alg: No test for adler32 (adler32-zlib) [12551.607701] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [12551.974624] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12553.679356] Key type lgssc registered [12555.015573] Lustre: Echo OBD driver; http://www.lustre.org/ [12564.122317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12564.161905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12565.693103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12569.535697] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12575.201421] LustreError: 401218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12575.215729] LustreError: 401218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12578.733862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.040370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12583.390859] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12589.136964] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12592.874841] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12600.072595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12600.491489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12606.530359] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12610.737552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12612.850491] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12619.068927] Lustre: server umount lustre-MDT0001 complete [12619.850244] LustreError: 401213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.257589] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12620.264743] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12621.292324] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12621.296423] LustreError: 401929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12621.316441] LustreError: 401929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12626.401602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12626.407749] LustreError: 401218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12626.412110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12626.413885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12631.522158] LustreError: 401217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12631.528498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12631.979846] Lustre: server umount lustre-OST0000 complete [12635.960458] Lustre: server umount lustre-MDT0000 complete [12647.803698] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12650.274257] Key type lgssc unregistered [12650.579454] LNet: 404553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12650.586753] LNetError: 404553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12650.598944] LNet: Removed LNI 192.168.206.138@tcp [12651.413273] Key type .llcrypt unregistered [12651.423047] Key type ._llcrypt unregistered [12675.459995] Key type ._llcrypt registered [12675.462492] Key type .llcrypt registered [12675.563329] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [12691.124651] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12692.443036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12692.478157] alg: No test for adler32 (adler32-zlib) [12693.529118] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [12693.798410] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12695.479479] Key type lgssc registered [12696.478257] Lustre: Echo OBD driver; http://www.lustre.org/ [12735.534101] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12750.664781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12750.715889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.027555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12752.064405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12752.156977] Lustre: lustre-MDT0000: new disk, initializing [12752.325630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12752.347156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12757.103649] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12772.537826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12772.866889] Lustre: lustre-OST0000: new disk, initializing [12772.876127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12772.889499] Lustre: Skipped 1 previous similar message [12772.946826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12779.183594] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12780.622656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12780.628186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12780.650790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12786.954369] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12801.537829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12801.652410] Lustre: 408519:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12801.715073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12801.826732] Lustre: lustre-MDT0001: new disk, initializing [12801.959433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12801.990611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12801.994745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12806.643784] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [12815.693764] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12816.021047] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12821.493159] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12821.838891] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12829.287705] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.38@tcp (stopping) [12831.213283] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12831.267505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12833.726225] Lustre: server umount lustre-MDT0001 complete [12839.469961] LustreError: 408526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12839.500580] LustreError: 408526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12841.447673] LustreError: 409931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12841.478940] LustreError: 409931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12846.573926] LustreError: 408530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12846.596360] LustreError: 408530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12851.688434] LustreError: 409931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.714894] LustreError: 409931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12856.804824] LustreError: 408531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12856.806238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.857817] Lustre: Skipped 1 previous similar message [12856.866189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12856.875573] Lustre: Skipped 1 previous similar message [12859.382308] Lustre: server umount lustre-OST0000 complete [12870.165546] Lustre: server umount lustre-MDT0000 complete [12885.060850] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12888.566524] Key type lgssc unregistered [12889.231640] LNet: 413088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12889.253635] LNetError: 413088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12889.292260] LNet: Removed LNI 192.168.206.138@tcp [12890.912267] Key type .llcrypt unregistered [12890.915298] Key type ._llcrypt unregistered [12937.062522] Key type ._llcrypt registered [12937.065996] Key type .llcrypt registered [12937.211356] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [12937.569270] Key type .llcrypt unregistered [12937.572271] Key type ._llcrypt unregistered [12954.539994] Key type ._llcrypt registered [12954.543787] Key type .llcrypt registered [12954.677686] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [12955.492240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12955.585788] alg: No test for adler32 (adler32-zlib) [12956.534256] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [12956.720630] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [12958.399518] Key type lgssc registered [12959.631615] Lustre: Echo OBD driver; http://www.lustre.org/ [12987.245748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12987.271454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12988.589402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12988.657857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12988.821684] Lustre: lustre-MDT0000: new disk, initializing [12988.930215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12988.977912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12992.538260] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13002.778733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13002.913320] Lustre: 416443:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13002.955754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13002.958572] Lustre: Skipped 1 previous similar message [13003.011265] Lustre: lustre-MDT0001: new disk, initializing [13003.078340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13003.110235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13003.129165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13008.302833] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13016.214616] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13020.762799] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13027.278933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13027.605120] Lustre: lustre-OST0000: new disk, initializing [13027.609503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13027.708059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13033.653458] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13036.632901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13036.650165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13036.819403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13041.762330] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13049.315721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13049.333981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13049.355690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13052.389864] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13052.390619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13052.411374] Lustre: Skipped 1 previous similar message [13052.416663] Lustre: Skipped 1 previous similar message [13054.818907] Lustre: server umount lustre-MDT0000 complete [13058.678530] LustreError: 416436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775207020 with bad export cookie 12450430731472994973 [13058.697628] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13058.705626] LustreError: 416436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13059.206047] Lustre: server umount lustre-MDT0001 complete [13074.576290] Lustre: server umount lustre-OST0000 complete [13086.580670] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:04:06 (1775207046) [13094.540666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13094.911314] LustreError: 420099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13094.930638] LustreError: 420099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13094.983908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13099.749975] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13100.005097] LustreError: 420100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13105.132924] LustreError: 420099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13108.471828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13108.871849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13113.140409] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13119.726644] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13124.363619] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13145.269797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13145.530352] Lustre: Found index 0 for lustre-OST0000, updating log [13145.710517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13152.762367] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13159.040960] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13165.169699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13179.881712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13179.895813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13179.928503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13179.937754] Lustre: Skipped 1 previous similar message [13182.381657] Lustre: server umount lustre-OST0000 complete [13200.867580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13200.873953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13200.891201] Lustre: Skipped 1 previous similar message [13200.896202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13200.900652] Lustre: Skipped 3 previous similar messages [13204.805857] Lustre: server umount lustre-MDT0000 complete [13206.508484] LustreError: 420099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13206.524145] LustreError: 420099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13206.623239] LNet: 1 local NIs in recovery (showing 1): 192.168.206.138@tcp [13208.286790] LustreError: 420082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775207170 with bad export cookie 12450430731472996366 [13208.289744] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13208.296578] LustreError: 420082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13208.598497] Lustre: server umount lustre-MDT0001 complete [13218.391734] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:06:18 (1775207178) [13842.181128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13842.529917] LustreError: 426198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13842.552020] LustreError: 426198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13842.613981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13847.859220] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13848.034363] LustreError: 426199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13853.164518] LustreError: 426198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13858.160467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13858.626832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13863.762982] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13870.532735] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13875.207373] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13897.210519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13897.446679] Lustre: Found index 0 for lustre-OST0000, updating log [13897.679332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13897.823166] LNet: 1 local NIs in recovery (showing 1): 192.168.206.138@tcp [13905.398954] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [13913.174440] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13917.152465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13917.156893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13917.162603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13917.176199] Lustre: Skipped 1 previous similar message [13917.183564] Lustre: Skipped 2 previous similar messages [13922.518583] Lustre: server umount lustre-OST0000 complete [13928.419661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13928.430778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13928.468484] Lustre: Skipped 1 previous similar message [13929.506368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13929.517581] Lustre: Skipped 2 previous similar messages [13931.554663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13931.557630] Lustre: Skipped 3 previous similar messages [13933.793554] Lustre: server umount lustre-MDT0000 complete [13934.642403] LustreError: 426199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13934.673125] LustreError: 426199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13937.989260] LustreError: 426180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775207899 with bad export cookie 12450430731472997451 [13937.999216] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13938.008929] LustreError: 426180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13938.345714] Lustre: server umount lustre-MDT0001 complete [13957.218974] LNet: 1 local NIs in recovery (showing 1): 192.168.206.138@tcp [13968.287326] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [13978.613565] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14015.856100] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:19:35 (1775207975) [14034.471715] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [14037.384922] Key type lgssc unregistered [14037.685720] LNet: 432425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14037.692871] LNetError: 432425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14157.546465] LNet: Removed LNI 192.168.206.138@tcp [14158.528300] Key type .llcrypt unregistered [14158.531361] Key type ._llcrypt unregistered [14177.128811] Key type ._llcrypt registered [14177.131626] Key type .llcrypt registered [14177.240550] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14178.323520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14178.575736] alg: No test for adler32 (adler32-zlib) [14179.662620] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [14179.991866] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [14181.687249] Key type lgssc registered [14182.910454] Lustre: Echo OBD driver; http://www.lustre.org/ [14187.485309] LNet: Added LNI 192.168.206.138@tcp100 [8/256/0/180] [14188.291087] LNet: Added LNI 192.168.206.138@tcp101 [8/256/0/180] [14189.080163] LNet: Added LNI 192.168.206.138@tcp102 [8/256/0/180] [14189.900503] LNet: Added LNI 192.168.206.138@tcp103 [8/256/0/180] [14190.698954] LNet: Added LNI 192.168.206.138@tcp104 [8/256/0/180] [14191.602979] LNet: Added LNI 192.168.206.138@tcp105 [8/256/0/180] [14192.514485] LNet: Added LNI 192.168.206.138@tcp106 [8/256/0/180] [14193.384833] LNet: Added LNI 192.168.206.138@tcp107 [8/256/0/180] [14194.281975] LNet: Added LNI 192.168.206.138@tcp108 [8/256/0/180] [14195.284053] LNet: Added LNI 192.168.206.138@tcp109 [8/256/0/180] [14196.263581] LNet: Added LNI 192.168.206.138@tcp110 [8/256/0/180] [14197.193917] LNet: Added LNI 192.168.206.138@tcp111 [8/256/0/180] [14198.096815] LNet: Added LNI 192.168.206.138@tcp112 [8/256/0/180] [14199.006835] LNet: Added LNI 192.168.206.138@tcp113 [8/256/0/180] [14199.816374] LNet: Added LNI 192.168.206.138@tcp114 [8/256/0/180] [14200.646434] LNet: Added LNI 192.168.206.138@tcp115 [8/256/0/180] [14201.509177] LNet: Added LNI 192.168.206.138@tcp116 [8/256/0/180] [14202.410232] LNet: Added LNI 192.168.206.138@tcp117 [8/256/0/180] [14203.232137] LNet: Added LNI 192.168.206.138@tcp118 [8/256/0/180] [14204.045804] LNet: Added LNI 192.168.206.138@tcp119 [8/256/0/180] [14204.937943] LNet: Added LNI 192.168.206.138@tcp120 [8/256/0/180] [14205.818341] LNet: Added LNI 192.168.206.138@tcp121 [8/256/0/180] [14206.816347] LNet: Added LNI 192.168.206.138@tcp122 [8/256/0/180] [14207.560889] LNet: Added LNI 192.168.206.138@tcp123 [8/256/0/180] [14208.345215] LNet: Added LNI 192.168.206.138@tcp124 [8/256/0/180] [14209.244756] LNet: Added LNI 192.168.206.138@tcp125 [8/256/0/180] [14210.063719] LNet: Added LNI 192.168.206.138@tcp126 [8/256/0/180] [14210.927097] LNet: Added LNI 192.168.206.138@tcp127 [8/256/0/180] [14211.797770] LNet: Added LNI 192.168.206.138@tcp128 [8/256/0/180] [14212.736268] LNet: Added LNI 192.168.206.138@tcp129 [8/256/0/180] [14213.615448] LNet: Added LNI 192.168.206.138@tcp130 [8/256/0/180] [14214.582198] LNet: Added LNI 192.168.206.138@tcp131 [8/256/0/180] [14215.590535] LNet: Added LNI 192.168.206.138@tcp132 [8/256/0/180] [14216.385752] LNet: Added LNI 192.168.206.138@tcp133 [8/256/0/180] [14217.273793] LNet: Added LNI 192.168.206.138@tcp134 [8/256/0/180] [14218.164586] LNet: Added LNI 192.168.206.138@tcp135 [8/256/0/180] [14225.981098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14226.011069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14227.193282] Lustre: 435443:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [14227.290115] LNetError: 432904:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [14227.297439] LNetError: Protocol error connecting to 192.168.206.138@tcp122 at host 192.168.206.138:988: is it running a compatible version of Lustre? [14227.485971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14227.515910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14227.659696] Lustre: lustre-MDT0000: new disk, initializing [14227.744496] Lustre: 435502:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [14227.750872] Lustre: 435502:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14227.767269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14227.783257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14228.138794] LNet: 6 local NIs in recovery (showing 6): 192.168.206.138@tcp122, 192.168.206.138@tcp125, 192.168.206.138@tcp126, 192.168.206.138@tcp124, 192.168.206.138@tcp118, 192.168.206.138@tcp123 [14228.150394] LNetError: 435472:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [14228.209795] LNetError: 435472:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 8 previous similar messages [14228.244529] LNetError: Protocol error connecting to 192.168.206.138@tcp108 at host 192.168.206.138:988: is it running a compatible version of Lustre? [14228.265524] LNetError: Skipped 8 previous similar messages [14232.628953] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14243.310527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14243.472260] Lustre: 435459:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14243.570335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14243.576668] Lustre: Skipped 1 previous similar message [14243.656597] Lustre: lustre-MDT0001: new disk, initializing [14243.689709] Lustre: 436248:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [14243.699583] Lustre: 436248:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14243.828619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14243.873773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14243.895056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14248.127610] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14252.657367] Lustre: 436657:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [14252.680034] Lustre: 436657:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14256.454657] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14259.877426] LNetError: 435477:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [14259.909389] LNetError: Protocol error connecting to 192.168.206.138@tcp121 at host 192.168.206.138:988: is it running a compatible version of Lustre? [14261.492477] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14263.011404] LNetError: 435464:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [14263.023767] LNetError: Protocol error connecting to 192.168.206.138@tcp119 at host 192.168.206.138:988: is it running a compatible version of Lustre? [14269.106961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14269.424501] Lustre: lustre-OST0000: new disk, initializing [14269.432211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14269.624985] Lustre: 437218:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [14269.637106] Lustre: 437218:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14269.718472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14271.229257] LNetError: 435467:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [14271.239874] LNetError: 435467:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 1 previous similar message [14271.278074] LNetError: Protocol error connecting to 192.168.206.138@tcp121 at host 192.168.206.138:988: is it running a compatible version of Lustre? [14271.307660] LNetError: Skipped 1 previous similar message [14276.094879] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14279.319138] Lustre: 437676:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [14279.343498] Lustre: 437676:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14279.436386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14279.465841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14279.640358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14283.445401] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14284.514664] LNetError: 435471:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [14284.524205] LNetError: 435471:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 2 previous similar messages [14284.547368] LNetError: Protocol error connecting to 192.168.206.138@tcp121 at host 192.168.206.138:988: is it running a compatible version of Lustre? [14284.563970] LNetError: Skipped 2 previous similar messages [14295.011425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14295.018328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14295.027986] LustreError: Skipped 1 previous similar message [14295.030138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14295.049357] Lustre: Skipped 1 previous similar message [14295.062090] Lustre: Skipped 1 previous similar message [14297.876974] Lustre: server umount lustre-OST0000 complete [14300.900931] LNetError: 435472:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [14300.913457] LNetError: 435472:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 3 previous similar messages [14300.923265] LNetError: Protocol error connecting to 192.168.206.138@tcp119 at host 192.168.206.138:988: is it running a compatible version of Lustre? [14300.937559] LNetError: Skipped 3 previous similar messages [14305.261544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14305.267702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14305.277177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14305.293550] Lustre: Skipped 1 previous similar message [14307.913168] Lustre: server umount lustre-MDT0000 complete [14311.523714] LustreError: 435452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775208273 with bad export cookie 4840707501692101842 [14311.531503] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14311.531663] LustreError: 435452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14311.902920] Lustre: server umount lustre-MDT0001 complete [14323.453136] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [14325.992094] Key type lgssc unregistered [14326.480850] LNet: 438995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14326.491877] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14326.497093] LNet: Removed LNI 192.168.206.138@tcp [14327.520070] LNet: Removed LNI 192.168.206.138@tcp100 [14327.525122] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14327.535636] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14327.545888] LNet: Removed LNI 192.168.206.138@tcp101 [14328.607541] LNet: Removed LNI 192.168.206.138@tcp102 [14328.611088] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14328.622812] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14328.638197] LNet: Removed LNI 192.168.206.138@tcp103 [14329.695171] LNet: Removed LNI 192.168.206.138@tcp104 [14330.719149] LNet: Removed LNI 192.168.206.138@tcp105 [14330.723893] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14330.730374] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14330.739088] LNet: Removed LNI 192.168.206.138@tcp106 [14331.743241] LNet: Removed LNI 192.168.206.138@tcp107 [14332.767824] LNet: Removed LNI 192.168.206.138@tcp108 [14333.791250] LNet: Removed LNI 192.168.206.138@tcp109 [14334.815576] LNet: Removed LNI 192.168.206.138@tcp110 [14334.820503] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14334.828712] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14334.842663] LNet: Removed LNI 192.168.206.138@tcp111 [14335.903161] LNet: Removed LNI 192.168.206.138@tcp112 [14336.929077] LNet: Removed LNI 192.168.206.138@tcp113 [14337.951340] LNet: Removed LNI 192.168.206.138@tcp114 [14338.975244] LNet: Removed LNI 192.168.206.138@tcp115 [14339.999271] LNet: Removed LNI 192.168.206.138@tcp116 [14341.023190] LNet: Removed LNI 192.168.206.138@tcp117 [14342.048464] LNet: Removed LNI 192.168.206.138@tcp118 [14343.071432] LNet: Removed LNI 192.168.206.138@tcp119 [14343.077067] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14343.093563] LNetError: 438995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14343.108846] LNet: Removed LNI 192.168.206.138@tcp120 [14344.159194] LNet: Removed LNI 192.168.206.138@tcp121 [14345.183660] LNet: Removed LNI 192.168.206.138@tcp122 [14346.207195] LNet: Removed LNI 192.168.206.138@tcp123 [14347.231204] LNet: Removed LNI 192.168.206.138@tcp124 [14348.255173] LNet: Removed LNI 192.168.206.138@tcp125 [14349.279165] LNet: Removed LNI 192.168.206.138@tcp126 [14349.282528] LNet: Removed LNI 192.168.206.138@tcp127 [14349.285992] LNet: Removed LNI 192.168.206.138@tcp128 [14349.291084] LNet: Removed LNI 192.168.206.138@tcp129 [14349.294756] LNet: Removed LNI 192.168.206.138@tcp130 [14349.297475] LNet: Removed LNI 192.168.206.138@tcp131 [14349.305167] LNet: Removed LNI 192.168.206.138@tcp132 [14349.313843] LNet: Removed LNI 192.168.206.138@tcp133 [14349.316642] LNet: Removed LNI 192.168.206.138@tcp134 [14349.331176] LNet: Removed LNI 192.168.206.138@tcp135 [14350.064703] Key type .llcrypt unregistered [14350.066366] Key type ._llcrypt unregistered [14373.274396] Key type ._llcrypt registered [14373.279254] Key type .llcrypt registered [14373.447277] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [14387.053612] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14387.987652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14388.020345] alg: No test for adler32 (adler32-zlib) [14388.971336] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [14389.130849] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [14390.815260] Key type lgssc registered [14392.315575] Lustre: Echo OBD driver; http://www.lustre.org/ [14427.030330] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:26:26 (1775208386) [14447.335614] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [14450.459515] Key type lgssc unregistered [14450.757793] LNet: 443032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14450.766284] LNetError: 443032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14450.786565] LNet: Removed LNI 192.168.206.138@tcp [14451.626688] Key type .llcrypt unregistered [14451.630494] Key type ._llcrypt unregistered [14469.034747] Key type ._llcrypt registered [14469.043414] Key type .llcrypt registered [14469.175802] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14470.278592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14470.410092] alg: No test for adler32 (adler32-zlib) [14471.511183] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [14471.894765] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [14473.583810] Key type lgssc registered [14474.852841] Lustre: Echo OBD driver; http://www.lustre.org/ [14485.682312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14485.706161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14487.137196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14487.211372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14487.341226] Lustre: lustre-MDT0000: new disk, initializing [14487.423748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14487.455242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14492.326955] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14504.160506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14504.320386] Lustre: 444227:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14504.418651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14504.434851] Lustre: Skipped 1 previous similar message [14504.510851] Lustre: lustre-MDT0001: new disk, initializing [14504.613092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14504.661183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14504.674328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14509.261556] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14516.739703] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14521.020768] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14528.284780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14528.600733] Lustre: lustre-OST0000: new disk, initializing [14528.604698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14528.752402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14534.707557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14534.721630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14534.841462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14535.493779] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14543.468729] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14548.378209] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing lnet_if_list [14554.289657] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14554.316472] LNet: Added LNI 192.168.206.138@tcp10 [8/256/0/180] [14560.570845] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14560.598568] LNet: Added LNI 192.168.206.138@tcp11 [8/256/0/180] [14565.025469] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14565.062611] LNet: Added LNI 192.168.206.138@tcp12 [8/256/0/180] [14570.435457] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14570.498895] LNet: Added LNI 192.168.206.138@tcp13 [8/256/0/180] [14575.232958] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14575.285690] LNet: Added LNI 192.168.206.138@tcp14 [8/256/0/180] [14580.198465] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14580.263364] LNet: Added LNI 192.168.206.138@tcp15 [8/256/0/180] [14585.806141] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14585.863256] LNet: Added LNI 192.168.206.138@tcp16 [8/256/0/180] [14590.716119] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14590.783659] LNet: Added LNI 192.168.206.138@tcp17 [8/256/0/180] [14595.943464] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14595.969704] LNet: Added LNI 192.168.206.138@tcp18 [8/256/0/180] [14602.003613] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14602.035568] LNet: Added LNI 192.168.206.138@tcp19 [8/256/0/180] [14608.543366] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14608.637934] LNet: Added LNI 192.168.206.138@tcp20 [8/256/0/180] [14616.105301] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14616.156795] LNet: Added LNI 192.168.206.138@tcp21 [8/256/0/180] [14622.746129] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14622.775330] LNet: Added LNI 192.168.206.138@tcp22 [8/256/0/180] [14628.776370] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14628.805209] LNet: Added LNI 192.168.206.138@tcp23 [8/256/0/180] [14635.803967] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14635.868628] LNet: Added LNI 192.168.206.138@tcp24 [8/256/0/180] [14643.238109] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14643.308882] LNet: Added LNI 192.168.206.138@tcp25 [8/256/0/180] [14649.216906] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14649.269314] LNet: Added LNI 192.168.206.138@tcp26 [8/256/0/180] [14656.504175] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14656.548407] LNet: Added LNI 192.168.206.138@tcp27 [8/256/0/180] [14663.382110] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14663.418319] LNet: Added LNI 192.168.206.138@tcp28 [8/256/0/180] [14670.441342] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14670.468501] LNet: Added LNI 192.168.206.138@tcp29 [8/256/0/180] [14677.832648] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14677.860067] LNet: Added LNI 192.168.206.138@tcp30 [8/256/0/180] [14684.544205] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14684.578812] LNet: Added LNI 192.168.206.138@tcp31 [8/256/0/180] [14691.761487] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14691.783852] LNet: Added LNI 192.168.206.138@tcp32 [8/256/0/180] [14698.444963] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14698.478620] LNet: Added LNI 192.168.206.138@tcp33 [8/256/0/180] [14705.520832] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14705.557897] LNet: Added LNI 192.168.206.138@tcp34 [8/256/0/180] [14712.969800] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14713.015743] LNet: Added LNI 192.168.206.138@tcp35 [8/256/0/180] [14720.725772] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14720.788195] LNet: Added LNI 192.168.206.138@tcp36 [8/256/0/180] [14728.719403] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14728.752749] LNet: Added LNI 192.168.206.138@tcp37 [8/256/0/180] [14736.470111] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14736.500943] LNet: Added LNI 192.168.206.138@tcp38 [8/256/0/180] [14744.384800] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14744.440527] LNet: Added LNI 192.168.206.138@tcp39 [8/256/0/180] [14752.361894] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14752.393991] LNet: Added LNI 192.168.206.138@tcp40 [8/256/0/180] [14759.900613] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14759.932783] LNet: Added LNI 192.168.206.138@tcp41 [8/256/0/180] [14767.669088] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14767.717149] LNet: Added LNI 192.168.206.138@tcp42 [8/256/0/180] [14775.792883] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14775.833608] LNet: Added LNI 192.168.206.138@tcp43 [8/256/0/180] [14783.229435] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14783.265464] LNet: Added LNI 192.168.206.138@tcp44 [8/256/0/180] [14790.766404] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14790.798618] LNet: Added LNI 192.168.206.138@tcp45 [8/256/0/180] [14802.562391] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14802.593897] LNet: Added LNI 192.168.206.138@tcp46 [8/256/0/180] [14812.656305] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14812.732348] LNet: Added LNI 192.168.206.138@tcp47 [8/256/0/180] [14820.986853] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14821.006538] LNet: Added LNI 192.168.206.138@tcp48 [8/256/0/180] [14830.338466] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14830.392869] LNet: Added LNI 192.168.206.138@tcp49 [8/256/0/180] [14837.953750] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14838.000414] LNet: Added LNI 192.168.206.138@tcp50 [8/256/0/180] [14854.784110] Lustre: server umount lustre-MDT0000 complete [14857.699816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14857.719092] Lustre: Skipped 1 previous similar message [14858.201156] LustreError: 444221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775208820 with bad export cookie 9166568172184597982 [14858.203659] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14858.210977] LustreError: 444221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14860.654632] Lustre: server umount lustre-MDT0001 complete [14878.179282] Lustre: 443501:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775208824/real 1775208824] req@000000006ad2089b x1861440958013312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775208840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14878.213296] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14878.229314] Lustre: Skipped 1 previous similar message [14879.201256] Lustre: 443500:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775208825/real 1775208825] req@0000000039fb5545 x1861440958013440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775208841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14879.997784] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [14883.296180] Lustre: 443502:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775208829/real 1775208829] req@0000000047fd1a6a x1861440958013568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775208845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14889.439099] Lustre: 443500:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775208835/real 1775208835] req@000000003adc9cd2 x1861440958013952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775208851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14889.466725] Lustre: 443500:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14889.627988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14893.599193] Lustre: 443500:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775208839/real 1775208839] req@0000000027983a10 x1861440958014208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775208855 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14904.209919] LustreError: 445966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14904.225764] LustreError: 445966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14904.311642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14908.323703] LustreError: 445967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14908.493851] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14913.511984] LustreError: 445966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14918.628956] LustreError: 445966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14918.672275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14919.147280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14922.226412] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14923.315339] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14924.063650] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [14930.490646] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14935.008711] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14959.080613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14959.097753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14959.102990] Lustre: Skipped 1 previous similar message [14964.194397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14964.203616] Lustre: Skipped 1 previous similar message [14969.316238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14969.324483] Lustre: Skipped 1 previous similar message [14969.823207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14970.169825] Lustre: server umount lustre-OST0000 complete [14975.456944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14975.471988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14975.492627] Lustre: Skipped 1 previous similar message [14975.502389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14979.554039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14979.557884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14980.939052] Lustre: server umount lustre-MDT0000 complete [14984.673386] LustreError: 469489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14984.693950] LustreError: 469489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14984.917216] LustreError: 444221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775208946 with bad export cookie 9166568172184600551 [14984.935877] LustreError: 444221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14984.942695] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14985.236548] Lustre: server umount lustre-MDT0001 complete [14996.437200] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [14999.438314] Key type lgssc unregistered [14999.825752] LNet: 472045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14999.834027] LNetError: 472045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14999.842550] LNet: Removed LNI 192.168.206.138@tcp [14999.846324] LNet: Removed LNI 192.168.206.138@tcp10 [14999.849504] LNet: Removed LNI 192.168.206.138@tcp11 [14999.860952] LNet: Removed LNI 192.168.206.138@tcp12 [14999.864785] LNet: Removed LNI 192.168.206.138@tcp13 [14999.872305] LNet: Removed LNI 192.168.206.138@tcp14 [14999.882851] LNet: Removed LNI 192.168.206.138@tcp15 [14999.886444] LNet: Removed LNI 192.168.206.138@tcp16 [14999.900482] LNet: Removed LNI 192.168.206.138@tcp17 [14999.907442] LNet: Removed LNI 192.168.206.138@tcp18 [14999.916192] LNet: Removed LNI 192.168.206.138@tcp19 [14999.927793] LNet: Removed LNI 192.168.206.138@tcp20 [14999.944407] LNet: Removed LNI 192.168.206.138@tcp21 [14999.950311] LNet: Removed LNI 192.168.206.138@tcp22 [14999.961283] LNet: Removed LNI 192.168.206.138@tcp23 [14999.968387] LNet: Removed LNI 192.168.206.138@tcp24 [14999.972873] LNet: Removed LNI 192.168.206.138@tcp25 [14999.983369] LNet: Removed LNI 192.168.206.138@tcp26 [14999.994155] LNet: Removed LNI 192.168.206.138@tcp27 [14999.996800] LNet: Removed LNI 192.168.206.138@tcp28 [15000.003960] LNet: Removed LNI 192.168.206.138@tcp29 [15000.014553] LNet: Removed LNI 192.168.206.138@tcp30 [15000.017947] LNet: Removed LNI 192.168.206.138@tcp31 [15000.030452] LNet: Removed LNI 192.168.206.138@tcp32 [15000.035390] LNet: Removed LNI 192.168.206.138@tcp33 [15000.039601] LNet: Removed LNI 192.168.206.138@tcp34 [15000.043821] LNet: Removed LNI 192.168.206.138@tcp35 [15000.048299] LNet: Removed LNI 192.168.206.138@tcp36 [15000.052430] LNet: Removed LNI 192.168.206.138@tcp37 [15000.056460] LNet: Removed LNI 192.168.206.138@tcp38 [15000.060581] LNet: Removed LNI 192.168.206.138@tcp39 [15000.065742] LNet: Removed LNI 192.168.206.138@tcp40 [15000.068963] LNet: Removed LNI 192.168.206.138@tcp41 [15000.071948] LNet: Removed LNI 192.168.206.138@tcp42 [15000.075334] LNet: Removed LNI 192.168.206.138@tcp43 [15000.077097] LNet: Removed LNI 192.168.206.138@tcp44 [15000.080352] LNet: Removed LNI 192.168.206.138@tcp45 [15000.082645] LNet: Removed LNI 192.168.206.138@tcp46 [15000.084532] LNet: Removed LNI 192.168.206.138@tcp47 [15000.086080] LNet: Removed LNI 192.168.206.138@tcp48 [15000.088157] LNet: Removed LNI 192.168.206.138@tcp49 [15000.097838] LNet: Removed LNI 192.168.206.138@tcp50 [15000.970615] Key type .llcrypt unregistered [15000.972569] Key type ._llcrypt unregistered [15020.515159] Key type ._llcrypt registered [15020.527773] Key type .llcrypt registered [15020.633812] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [15021.435418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15021.464801] alg: No test for adler32 (adler32-zlib) [15022.548925] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [15022.819544] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [15024.495186] Key type lgssc registered [15025.877776] Lustre: Echo OBD driver; http://www.lustre.org/ [15033.589878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15033.608347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15035.102180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15039.682801] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [15044.582349] LustreError: 473292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15044.599678] LustreError: 473292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15048.666846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15049.165889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15053.850754] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [15059.420209] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15063.689320] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15071.017776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15071.331407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15077.615044] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [15079.535026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [15083.129753] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15093.240972] loop0: detected capacity change from 0 to 204800000 [15093.496846] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15093.510600] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15093.524856] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15099.593091] loop0: detected capacity change from 0 to 204800000 [15099.881683] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15099.897272] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15099.909900] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15105.611541] loop0: detected capacity change from 0 to 204800000 [15111.831716] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15111.946433] Lustre: fs15246-OST0000: new disk, initializing [15111.951381] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15112.015868] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15118.222944] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [15126.738850] loop1: detected capacity change from 0 to 204800000 [15131.767544] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15131.897881] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15131.952700] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15132.008314] Lustre: fs15246-MDT0000: new disk, initializing [15132.154483] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15132.196868] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15132.263721] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15133.280263] Lustre: 477653:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000015b56403 x1861441535651456/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775209111 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15135.535780] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [15142.391845] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15142.409963] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15142.448743] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15159.277736] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15159.290670] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15163.021949] Lustre: server umount fs15246-MDT0000 complete [15164.399027] LustreError: 474905:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15169.520706] LustreError: 473289:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15171.930802] Lustre: server umount fs15246-OST0000 complete [15184.865319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15184.870475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15184.905982] Lustre: Skipped 1 previous similar message [15184.934034] Lustre: Skipped 1 previous similar message [15189.993479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15190.007052] Lustre: Skipped 1 previous similar message [15190.543690] Lustre: server umount lustre-OST0000 complete [15197.668935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15197.680216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15197.705684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15200.229460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15201.724482] Lustre: server umount lustre-MDT0000 complete [15205.345763] LustreError: 474003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15205.368726] LustreError: 474003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15205.939665] LustreError: 473272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775209167 with bad export cookie 17815604950577255911 [15205.945133] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15205.956973] LustreError: 473272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15206.283511] Lustre: server umount lustre-MDT0001 complete [15219.949339] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [15224.451964] Key type lgssc unregistered [15224.950853] LNet: 480719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15224.956978] LNetError: 480719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15224.999284] LNet: Removed LNI 192.168.206.138@tcp [15226.068865] Key type .llcrypt unregistered [15226.076191] Key type ._llcrypt unregistered [15245.614992] Key type ._llcrypt registered [15245.625423] Key type .llcrypt registered [15245.772215] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [15246.614146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15246.629988] alg: No test for adler32 (adler32-zlib) [15247.653161] Lustre: Lustre: Build Version: 2.17.51_74_g48b5ed2 [15247.873237] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [15249.503182] Key type lgssc registered [15251.186733] Lustre: Echo OBD driver; http://www.lustre.org/ [15296.702648] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [15307.551178] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [15344.968551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15345.029460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15346.384273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15346.446286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15346.543977] Lustre: lustre-MDT0000: new disk, initializing [15346.645938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15346.677951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15350.786254] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [15360.990544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15361.126431] Lustre: 485137:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15361.180839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15361.184451] Lustre: Skipped 1 previous similar message [15361.262304] Lustre: lustre-MDT0001: new disk, initializing [15361.374532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15361.410678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15361.429088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15365.335688] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [15372.562253] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15376.438799] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15383.331462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15383.607746] Lustre: lustre-OST0000: new disk, initializing [15383.611394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15383.732380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15389.791567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15389.812395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15389.951603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15390.049853] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [15398.518726] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15403.328122] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15403.597529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15408.254974] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15408.528862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15410.658288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15410.659930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15410.664917] LustreError: Skipped 1 previous similar message [15410.666499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15410.686214] Lustre: Skipped 1 previous similar message [15410.705341] Lustre: Skipped 1 previous similar message [15415.778172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15415.785218] Lustre: Skipped 1 previous similar message [15416.454312] Lustre: server umount lustre-OST0000 complete [15420.897993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15420.927585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15420.941235] Lustre: Skipped 1 previous similar message [15426.016247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15426.020991] Lustre: Skipped 1 previous similar message [15426.290854] Lustre: server umount lustre-MDT0000 complete [15429.896333] LustreError: 485129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775209391 with bad export cookie 1420027486187227602 [15429.899423] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15429.915419] LustreError: 485129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15430.274366] Lustre: server umount lustre-MDT0001 complete [15455.368565] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [15465.149840] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [15496.295748] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15204 sec ========= 05:44:16 (1775209456) [15497.979982] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:44:17 (1775209457) === [15500.914237] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:44:21 (1775209461) === [15539.293504] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [15542.150448] Key type lgssc unregistered [15542.437957] LNet: 493326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15542.450391] LNetError: 493326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15542.472795] LNet: Removed LNI 192.168.206.138@tcp [15543.398511] Key type .llcrypt unregistered [15543.400872] Key type ._llcrypt unregistered