[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 620214192 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002439] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.012178] LSM: Security Framework initializing [ 0.013051] Yama: becoming mindful. [ 0.014056] SELinux: Initializing. [ 0.015195] *** VALIDATE selinux *** [ 0.026000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031214] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034194] *** VALIDATE tmpfs *** [ 0.037184] *** VALIDATE proc *** [ 0.039381] *** VALIDATE cgroup *** [ 0.040008] *** VALIDATE cgroup2 *** [ 0.042205] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045024] Spectre V2 : User space: Vulnerable [ 0.046005] Speculative Store Bypass: Vulnerable [ 0.049573] debug: unmapping init [mem 0xffffffffb6c59000-0xffffffffb6c60fff] [ 0.051357] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052690] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053030] ... version: 2 [ 0.054012] ... bit width: 48 [ 0.055021] ... generic registers: 4 [ 0.056011] ... value mask: 0000ffffffffffff [ 0.057013] ... max period: 00007fffffffffff [ 0.058010] ... fixed-purpose events: 3 [ 0.059023] ... event mask: 000000070000000f [ 0.061072] rcu: Hierarchical SRCU implementation. [ 0.063821] smp: Bringing up secondary CPUs ... [ 0.064898] x86: Booting SMP configuration: [ 0.065032] .... node #0, CPUs: #1 #2 #3 [ 0.080377] smp: Brought up 1 node, 4 CPUs [ 0.082019] smpboot: Max logical packages: 1 [ 0.083011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141377] node 0 deferred pages initialised in 56ms [ 0.146032] devtmpfs: initialized [ 0.147224] x86/mm: Memory block size: 128MB [ 0.152119] gcov: version magic: 0x41383552 [ 0.156194] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158284] pinctrl core: initialized pinctrl subsystem [ 0.159355] [ 0.159898] ************************************************************* [ 0.160013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161010] ** ** [ 0.162012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163013] ** ** [ 0.164012] ** This means that this kernel is built to expose internal ** [ 0.165012] ** IOMMU data structures, which may compromise security on ** [ 0.166011] ** your system. ** [ 0.167014] ** ** [ 0.168015] ** If you see this message and you are not debugging the ** [ 0.169012] ** kernel, report this immediately to your vendor! ** [ 0.170014] ** ** [ 0.171014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172013] ************************************************************* [ 0.216183] NET: Registered protocol family 16 [ 0.220842] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.230074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.234357] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.238115] cpuidle: using governor menu [ 0.240000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.244741] PCI: Using configuration type 1 for base access [ 0.247162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.255410] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.257017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.259084] cryptd: max_cpu_qlen set to 1000 [ 0.261222] ACPI: Added _OSI(Module Device) [ 0.263028] ACPI: Added _OSI(Processor Device) [ 0.264020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.266015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.273075] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.279444] ACPI: Interpreter enabled [ 0.281053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.283056] ACPI: Using IOAPIC for interrupt routing [ 0.285112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.289491] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.303848] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.306036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.309018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.313178] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.318672] acpiphp: Slot [2] registered [ 0.320152] acpiphp: Slot [5] registered [ 0.322232] acpiphp: Slot [6] registered [ 0.324085] acpiphp: Slot [7] registered [ 0.325242] acpiphp: Slot [8] registered [ 0.327598] acpiphp: Slot [9] registered [ 0.329168] acpiphp: Slot [10] registered [ 0.332157] acpiphp: Slot [3] registered [ 0.333000] acpiphp: Slot [4] registered [ 0.333000] acpiphp: Slot [11] registered [ 0.333102] acpiphp: Slot [12] registered [ 0.334000] acpiphp: Slot [13] registered [ 0.337102] acpiphp: Slot [14] registered [ 0.338091] acpiphp: Slot [15] registered [ 0.340266] acpiphp: Slot [16] registered [ 0.342206] acpiphp: Slot [17] registered [ 0.345187] acpiphp: Slot [18] registered [ 0.347123] acpiphp: Slot [19] registered [ 0.348262] acpiphp: Slot [20] registered [ 0.350116] acpiphp: Slot [21] registered [ 0.352104] acpiphp: Slot [22] registered [ 0.353235] acpiphp: Slot [23] registered [ 0.355082] acpiphp: Slot [24] registered [ 0.357115] acpiphp: Slot [25] registered [ 0.359116] acpiphp: Slot [26] registered [ 0.360165] acpiphp: Slot [27] registered [ 0.362100] acpiphp: Slot [28] registered [ 0.363113] acpiphp: Slot [29] registered [ 0.365201] acpiphp: Slot [30] registered [ 0.367106] acpiphp: Slot [31] registered [ 0.368061] PCI host bridge to bus 0000:00 [ 0.370017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.373025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.375027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.377025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.380019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.382023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.384373] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.388092] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.391435] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.406019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.412060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.414020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.417018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.425022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.429674] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.434886] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.439052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.442821] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.446000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.459015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.465016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.474009] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.500062] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.506052] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.532016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.543534] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.554019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.564017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.588017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.676616] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.695023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.703015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.712021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.734049] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.748018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.760019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.784015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.798095] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.808018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.819023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.837013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.850786] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.861019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.872017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.898024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.910570] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.913404] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.916802] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.919491] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.923231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.929171] iommu: Default domain type: Passthrough [ 0.930762] SCSI subsystem initialized [ 0.933179] ACPI: bus type USB registered [ 0.934121] usbcore: registered new interface driver usbfs [ 0.937098] usbcore: registered new interface driver hub [ 0.940115] usbcore: registered new device driver usb [ 0.942167] pps_core: LinuxPPS API ver. 1 registered [ 0.943007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.947177] PTP clock support registered [ 0.950066] EDAC MC: Ver: 3.0.0 [ 0.952113] PCI: Using ACPI for IRQ routing [ 0.954401] NetLabel: Initializing [ 0.956011] NetLabel: domain hash size = 128 [ 0.958012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.961094] NetLabel: unlabeled traffic allowed by default [ 0.964471] vgaarb: loaded [ 0.966769] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.969017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.979223] clocksource: Switched to clocksource kvm-clock [ 1.120482] VFS: Disk quotas dquot_6.6.0 [ 1.122355] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.125180] *** VALIDATE ramfs *** [ 1.126516] *** VALIDATE hugetlbfs *** [ 1.128871] pnp: PnP ACPI init [ 1.131762] pnp: PnP ACPI: found 6 devices [ 1.148711] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.152263] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.154750] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.156896] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.159520] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.162105] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.165310] NET: Registered protocol family 2 [ 1.167692] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.172588] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.175894] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.181498] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.185155] TCP: Hash tables configured (established 65536 bind 65536) [ 1.188742] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.192427] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.196342] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.199784] NET: Registered protocol family 1 [ 1.202307] RPC: Registered named UNIX socket transport module. [ 1.204626] RPC: Registered udp transport module. [ 1.206865] RPC: Registered tcp transport module. [ 1.208623] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.210996] NET: Registered protocol family 44 [ 1.212787] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.215135] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.217355] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.219842] PCI: CLS 0 bytes, default 64 [ 1.221600] Unpacking initramfs... [ 2.969997] debug: unmapping init [mem 0xffff8a353cc54000-0xffff8a353ffbffff] [ 2.973912] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.978516] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.981397] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.129687] Initialise system trusted keyrings [ 4.131024] Key type blacklist registered [ 4.132626] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.140950] zbud: loaded [ 4.143647] *** VALIDATE nfs *** [ 4.144697] *** VALIDATE nfs4 *** [ 4.145891] pstore: using deflate compression [ 4.149268] Platform Keyring initialized [ 4.275604] NET: Registered protocol family 38 [ 4.278529] Key type asymmetric registered [ 4.280517] Asymmetric key parser 'x509' registered [ 4.282843] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.285341] io scheduler mq-deadline registered [ 4.286751] io scheduler kyber registered [ 4.295331] io scheduler bfq registered [ 4.296946] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.299693] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.302501] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.305185] ACPI: Power Button [PWRF] [ 4.310298] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.317205] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.327858] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.333434] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.355377] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.387322] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.417100] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.421628] Non-volatile memory driver v1.3 [ 4.423101] Linux agpgart interface v0.103 [ 4.451895] virtio_blk virtio1: [vda] 146544 512-byte logical blocks (75.0 MB/71.6 MiB) [ 4.454566] vda: detected capacity change from 0 to 75030528 [ 4.470306] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.473365] vdb: detected capacity change from 0 to 1073741824 [ 4.489370] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.491741] vdc: detected capacity change from 0 to 2621440000 [ 4.507691] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.510057] vdd: detected capacity change from 0 to 2621440000 [ 4.527464] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.529950] vde: detected capacity change from 0 to 4294967296 [ 4.550490] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.553024] vdf: detected capacity change from 0 to 4294967296 [ 4.564990] libphy: Fixed MDIO Bus: probed [ 4.574740] usbcore: registered new interface driver usbserial_generic [ 4.576947] usbserial: USB Serial support registered for generic [ 4.579286] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.583204] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.584913] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.587209] mousedev: PS/2 mouse device common for all mice [ 4.589586] rtc_cmos 00:05: RTC can wake from S4 [ 4.591949] rtc_cmos 00:05: registered as rtc0 [ 4.594189] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.596410] intel_pstate: CPU model not supported [ 4.599729] hid: raw HID events driver (C) Jiri Kosina [ 4.601663] usbcore: registered new interface driver usbhid [ 4.603407] usbhid: USB HID core driver [ 4.605470] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.608093] drop_monitor: Initializing network drop monitor service [ 4.608219] Initializing XFRM netlink socket [ 4.608566] NET: Registered protocol family 10 [ 4.611298] Segment Routing with IPv6 [ 4.611349] NET: Registered protocol family 17 [ 4.614269] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.617568] mpls_gso: MPLS GSO support [ 4.618909] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.630513] RAS: Correctable Errors collector initialized. [ 4.632588] AVX version of gcm_enc/dec engaged. [ 4.634288] AES CTR mode by8 optimization enabled [ 4.718524] sched_clock: Marking stable (4718506045, 0)->(6063310786, -1344804741) [ 4.722981] registered taskstats version 1 [ 4.725675] Loading compiled-in X.509 certificates [ 4.727644] zswap: loaded using pool lzo/zbud [ 4.760300] Key type big_key registered [ 4.774610] Key type encrypted registered [ 4.776694] ima: No TPM chip found, activating TPM-bypass! [ 4.779578] ima: Allocated hash algorithm: sha1 [ 4.781697] ima: No architecture policies found [ 4.798621] evm: Initialising EVM extended attributes: [ 4.800714] evm: security.selinux [ 4.801594] evm: security.ima [ 4.802297] evm: security.capability [ 4.803049] evm: HMAC attrs: 0x1 [ 4.806559] rtc_cmos 00:05: setting system clock to 2026-08-22 04:27:39 UTC (1787372859) [ 4.813365] debug: unmapping init [mem 0xffffffffb7c03000-0xffffffffb7dfffff] [ 4.816578] debug: unmapping init [mem 0xffffffffb6982000-0xffffffffb6c58fff] [ 4.821056] Write protecting the kernel read-only data: 28672k [ 4.824590] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 4.827478] debug: unmapping init [mem 0xffffffffb5914000-0xffffffffb59fffff] [ 4.862269] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.868717] systemd[1]: Detected virtualization kvm. [ 4.870125] systemd[1]: Detected architecture x86-64. [ 4.871558] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.897132] systemd[1]: No hostname configured. [ 4.898365] systemd[1]: Set hostname to . [ 4.899750] random: systemd: uninitialized urandom read (16 bytes read) [ 4.901605] systemd[1]: Initializing machine ID from random generator. [ 5.060657] random: systemd: uninitialized urandom read (16 bytes read) [ 5.062820] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.072323] random: systemd: uninitialized urandom read (16 bytes read) [ 5.074702] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.078207] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.234233] device-mapper: uevent: version 1.0.3 [ 6.237228] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 8.097903] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.199752] scsi host0: ata_piix [ 8.255767] scsi host1: ata_piix [ 8.260324] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.268967] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.294189] random: fast init done [ 13.327350] random: crng init done [ 13.330848] random: 7 urandom warning(s) missed due to ratelimiting [ 15.703299] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.123984] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.255034] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.897358] SELinux: Disabled at runtime. [ 21.098923] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.108882] systemd[1]: Detected virtualization kvm. [ 21.110680] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.288794] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.296422] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.306626] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.310963] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.314379] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.330615] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.339903] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 23.677750] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.266758] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.991768] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.040888] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.357029] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.575749] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 36.293780] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (14s / no limit)[ 36.981667] NFS: Registering the id_resolver key type [ 36.990205] Key type id_resolver registered [ 37.001377] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [* ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root support (17s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg243-server login: [ 69.616881] hrtimer: interrupt took 6867448 ns [ 152.612910] libcfs: loading out-of-tree module taints kernel. [ 152.656155] Key type ._llcrypt registered [ 152.658311] Key type .llcrypt registered [ 152.744445] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 168.606181] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 170.173863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 170.183507] alg: No test for adler32 (adler32-zlib) [ 171.638215] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 172.575337] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 174.383156] Key type lgssc registered [ 175.931431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 194.291946] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 237.285230] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 251.222834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 251.256928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.539153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 252.576855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 252.739376] Lustre: lustre-MDT0000: new disk, initializing [ 252.826844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.839608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 257.882376] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.643067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 271.727288] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 271.767748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 271.777107] Lustre: Skipped 1 previous similar message [ 271.874515] Lustre: lustre-MDT0001: new disk, initializing [ 271.944208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.008210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 272.039692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 276.768160] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.877433] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 292.242780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.721764] Lustre: lustre-OST0000: new disk, initializing [ 292.723792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 292.729108] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 292.809398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 299.068375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 299.071491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 299.149722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 299.708408] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 314.599911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 314.750213] Lustre: lustre-OST0001: new disk, initializing [ 314.756565] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 314.760883] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 314.800216] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 323.235753] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 324.143440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 324.150929] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 324.225276] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 337.339043] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 347.798950] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 355.466321] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing check_logdir /tmp/testlogs/ [ 361.642097] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing yml_node [ 366.596647] Lustre: DEBUG MARKER: Client: 2.17.57.47 [ 369.546123] Lustre: DEBUG MARKER: MDS: 2.17.57.47 [ 372.643333] Lustre: DEBUG MARKER: OSS: 2.17.57.47 [ 374.703637] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 00:33:48 EDT 2026 [ 398.038460] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 400.091651] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 420.322355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 420.325674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.353483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 421.344726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 421.346466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 421.391392] Lustre: Skipped 1 previous similar message [ 426.703568] Lustre: server umount lustre-MDT0000 complete [ 430.216163] LustreError: 9524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373284 with bad export cookie 5058528321794919194 [ 430.220508] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 430.243208] LustreError: 9524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 431.584940] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.585761] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.613551] Lustre: Skipped 1 previous similar message [ 431.614695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 431.614703] Lustre: Skipped 3 previous similar messages [ 431.655617] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 435.633069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 435.633068] LustreError: 10252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.654827] Lustre: Skipped 2 previous similar messages [ 436.961592] Lustre: server umount lustre-MDT0001 complete [ 442.781132] Lustre: server umount lustre-OST0000 complete [ 447.363821] Lustre: server umount lustre-OST0001 complete [ 456.572349] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 467.657423] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 508.343756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.636947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 508.659597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 508.713796] Lustre: lustre-MDT0000: new disk, initializing [ 508.764049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.782615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 513.071695] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 524.657969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.770279] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 524.850534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 524.858111] Lustre: Skipped 1 previous similar message [ 524.985676] Lustre: lustre-MDT0001: new disk, initializing [ 525.147613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 525.212709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 525.228199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 530.145714] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 541.412456] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 549.241643] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 557.085475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 557.384331] Lustre: lustre-OST0000: new disk, initializing [ 557.392733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 557.404322] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 557.480826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.054276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 559.072273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 559.183663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 564.096905] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 573.767711] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 578.374125] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 578.615490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 583.164449] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 583.454285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 589.795728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.805647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.811888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.815958] Lustre: Skipped 2 previous similar messages [ 589.833915] Lustre: Skipped 1 previous similar message [ 591.368200] Lustre: server umount lustre-OST0000 complete [ 594.913956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.032635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 600.043594] Lustre: Skipped 3 previous similar messages [ 609.252256] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 609.527960] Lustre: server umount lustre-MDT0000 complete [ 610.285666] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.319111] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 612.964331] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373467 with bad export cookie 5058528321794920727 [ 612.977636] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.980601] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 613.236430] Lustre: server umount lustre-MDT0001 complete [ 617.805768] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:37:51 (1787373471) [ 620.516916] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 622.194324] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:37:55 (1787373475) [ 629.463164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.143487] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.248602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.856498] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 635.365384] LustreError: 19466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.555330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.583122] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.617424] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 650.061950] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 658.576144] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 664.981449] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 673.042353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 680.509421] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 689.335785] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 700.896700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 700.916116] LustreError: Skipped 1 previous similar message [ 700.922833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.938374] Lustre: Skipped 1 previous similar message [ 700.953273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 700.958319] Lustre: Skipped 2 previous similar messages [ 706.165887] Lustre: server umount lustre-OST0000 complete [ 711.648098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 711.656356] LustreError: Skipped 1 previous similar message [ 716.486490] Lustre: server umount lustre-MDT0000 complete [ 716.780857] LustreError: 21154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.813613] LustreError: 21154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 721.521846] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373576 with bad export cookie 5058528321794921602 [ 721.528350] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.539764] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 721.957363] Lustre: server umount lustre-MDT0001 complete [ 736.381126] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 740.528501] Key type lgssc unregistered [ 740.940551] LNet: 23004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.948638] LNetError: 23004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.986748] LNet: Removed LNI 192.168.202.143@tcp [ 742.515326] Key type .llcrypt unregistered [ 742.517185] Key type ._llcrypt unregistered [ 768.054410] Key type ._llcrypt registered [ 768.059502] Key type .llcrypt registered [ 768.214931] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 769.799849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 769.826074] alg: No test for adler32 (adler32-zlib) [ 771.017941] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 771.360399] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 773.159175] Key type lgssc registered [ 774.610919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 783.195399] loop: module loaded [ 783.298681] loop0: detected capacity change from 0 to 204800000 [ 783.613561] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 783.627583] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 784.045727] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 790.077324] loop0: detected capacity change from 0 to 204800000 [ 790.343929] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 790.377681] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 790.399306] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 796.174697] loop0: detected capacity change from 0 to 204800000 [ 801.650682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 801.679644] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.010526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 803.109913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 803.222915] Lustre: lustre-MDT0000: new disk, initializing [ 803.293077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 803.314210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 808.473909] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 818.255667] loop1: detected capacity change from 0 to 204800000 [ 824.474696] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 824.716608] Lustre: lustre-OST07c6: new disk, initializing [ 824.721327] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 824.724910] Lustre: Skipped 1 previous similar message [ 824.729484] Lustre: 26192:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 824.794476] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 830.110597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 830.132882] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 830.190508] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 831.841978] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 836.947494] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 836.963473] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 836.971410] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 836.976668] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 836.985717] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 836.993848] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 856.744563] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 856.762386] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 856.770854] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 856.788375] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 856.804772] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 856.814742] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 861.875581] Lustre: 26199:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 861.887425] Lustre: 26199:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 861.894592] Lustre: 26199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 861.903891] Lustre: 26199:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.913631] Lustre: 26199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 861.921955] Lustre: 26199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.929839] Lustre: 26199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 861.938513] Lustre: 26199:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.948671] Lustre: 26199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 861.956958] Lustre: 26199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.971695] Lustre: 26199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 861.980613] Lustre: 26199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 865.140315] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 865.146941] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 865.159408] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 865.168495] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 865.172498] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 865.176417] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 865.179590] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 865.183737] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 865.186783] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 865.189581] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 865.192943] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 865.195662] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.308151] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 870.314365] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 870.320809] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 870.327658] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.333479] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 870.341205] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.347494] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 870.355605] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.362670] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 870.369586] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.376457] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 870.378988] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 878.674118] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 878.700777] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 878.717230] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 878.730840] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 878.742205] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 878.752702] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 878.760384] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 878.776439] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 878.788817] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 878.796048] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 878.805313] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 878.819192] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 896.882127] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 896.915682] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 896.928646] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 896.937182] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 896.944550] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 896.955561] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 896.969786] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 896.985556] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 896.990861] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 896.998169] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 897.011312] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 897.026829] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 932.832468] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.844450] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.867491] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 937.443497] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 938.795217] Lustre: server umount lustre-OST07c6 complete [ 948.100931] Lustre: server umount lustre-MDT0000 complete [ 966.016334] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 968.457489] Key type lgssc unregistered [ 968.777527] LNet: 28760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.786828] LNetError: 28760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.825843] LNet: Removed LNI 192.168.202.143@tcp [ 969.803443] Key type .llcrypt unregistered [ 969.806570] Key type ._llcrypt unregistered [ 988.790722] Key type ._llcrypt registered [ 988.795069] Key type .llcrypt registered [ 988.920273] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 990.118459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 990.277242] alg: No test for adler32 (adler32-zlib) [ 991.407190] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 991.560581] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 993.207209] Key type lgssc registered [ 994.447743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1002.303116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1002.345871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.884979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1008.712812] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1014.243645] LustreError: 30028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.257548] LustreError: 30028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1016.581315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.011476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.403567] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1029.901387] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1035.715796] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1043.073526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1043.628073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.757618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1051.186810] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1061.362128] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1077.736297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1077.745538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.747857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1077.751733] Lustre: Skipped 1 previous similar message [ 1077.760603] Lustre: Skipped 1 previous similar message [ 1081.503410] Lustre: server umount lustre-OST0000 complete [ 1087.975717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.984028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1087.990503] Lustre: Skipped 1 previous similar message [ 1088.001357] Lustre: Skipped 1 previous similar message [ 1091.678963] Lustre: server umount lustre-MDT0000 complete [ 1093.090180] LustreError: 30025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.128809] LustreError: 30025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1095.531624] LustreError: 30763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373950 with bad export cookie 2523914891885641652 [ 1095.537365] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1095.553238] LustreError: 30763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1095.895834] Lustre: server umount lustre-MDT0001 complete [ 1107.079540] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1109.786813] Key type lgssc unregistered [ 1110.197547] LNet: 33773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.210867] LNetError: 33773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.250542] LNet: Removed LNI 192.168.202.143@tcp [ 1111.175529] Key type .llcrypt unregistered [ 1111.178181] Key type ._llcrypt unregistered [ 1129.104328] Key type ._llcrypt registered [ 1129.107188] Key type .llcrypt registered [ 1129.213153] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1130.620411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1130.678979] alg: No test for adler32 (adler32-zlib) [ 1131.797694] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 1132.062584] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1133.815171] Key type lgssc registered [ 1135.323607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1144.378507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1144.397914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.774065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.298225] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1156.066842] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.109198] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1160.906311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.156896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1167.224748] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1175.742763] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1181.423117] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1188.347934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1188.829400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.949298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1195.447247] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1203.671318] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1215.978278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.978931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.993209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.999310] Lustre: Skipped 1 previous similar message [ 1219.958716] Lustre: server umount lustre-OST0000 complete [ 1226.214879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.223869] Lustre: Skipped 1 previous similar message [ 1226.228193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.232709] Lustre: Skipped 1 previous similar message [ 1230.072650] Lustre: server umount lustre-MDT0000 complete [ 1231.345077] LustreError: 35041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.359869] LustreError: 35041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1233.608678] LustreError: 35022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374088 with bad export cookie 13563877782511155605 [ 1233.620439] LustreError: 35022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1233.625405] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.107424] Lustre: server umount lustre-MDT0001 complete [ 1244.900465] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1247.577922] Key type lgssc unregistered [ 1247.873940] LNet: 38577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.894711] LNetError: 38577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.919901] LNet: Removed LNI 192.168.202.143@tcp [ 1248.810560] Key type .llcrypt unregistered [ 1248.813894] Key type ._llcrypt unregistered [ 1274.301644] Key type ._llcrypt registered [ 1274.304593] Key type .llcrypt registered [ 1274.437396] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1275.438617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1275.508628] alg: No test for adler32 (adler32-zlib) [ 1276.637518] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 1276.994070] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1278.719172] Key type lgssc registered [ 1280.111621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1287.988518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1288.009934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.413111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.816404] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1299.937416] LustreError: 39993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.954117] LustreError: 39993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1302.123438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1302.467398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1306.887189] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1313.736593] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1319.400714] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1326.429739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.748659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.195883] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1335.035493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1340.467607] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1347.326853] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.43@tcp (stopping) [ 1348.252906] Lustre: server umount lustre-MDT0000 complete [ 1348.580639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1348.590063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.600934] LustreError: 39993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.633620] LustreError: 41425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.645534] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.663402] LustreError: 41425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1351.517802] LustreError: 41835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374206 with bad export cookie 2312104606237452035 [ 1351.530769] LustreError: 41835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1351.532770] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1352.458993] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.43@tcp (stopping) [ 1355.743995] LustreError: 41232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.752811] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.767131] LustreError: 41232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1355.786246] Lustre: Skipped 1 previous similar message [ 1355.792870] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1357.958798] Lustre: server umount lustre-MDT0001 complete [ 1379.244571] Lustre: server umount lustre-OST0000 complete [ 1396.349905] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1399.003616] Key type lgssc unregistered [ 1399.303336] LNet: 43857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.311567] LNetError: 43857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.328869] LNet: Removed LNI 192.168.202.143@tcp [ 1400.238109] Key type .llcrypt unregistered [ 1400.239924] Key type ._llcrypt unregistered [ 1425.277457] Key type ._llcrypt registered [ 1425.280630] Key type .llcrypt registered [ 1425.363283] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1426.050171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1426.119069] alg: No test for adler32 (adler32-zlib) [ 1427.148730] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 1427.358074] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1429.031160] Key type lgssc registered [ 1430.077694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1438.276509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1438.308624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1439.816346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.830861] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1449.966857] LustreError: 45274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.991620] LustreError: 45274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1453.561802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.896066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.740583] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1464.358978] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1470.532599] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1477.325642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.545168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.986526] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1485.783170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1490.747899] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1497.879874] Lustre: lustre-OST0000: Not available for connect from 192.168.202.43@tcp (stopping) [ 1499.109748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1499.116228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.130532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.156529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.158921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1501.166758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.892248] Lustre: server umount lustre-OST0000 complete [ 1520.608738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1520.615153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.641335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.650372] Lustre: Skipped 1 previous similar message [ 1525.795746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1525.805390] Lustre: Skipped 3 previous similar messages [ 1526.443066] Lustre: server umount lustre-MDT0000 complete [ 1530.098236] LustreError: 45254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374384 with bad export cookie 4148599462572595532 [ 1530.114771] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.125358] LustreError: 45254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1536.612710] Lustre: server umount lustre-MDT0001 complete [ 1547.140541] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1549.693689] Key type lgssc unregistered [ 1549.949815] LNet: 48984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.966711] LNetError: 48984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.005771] LNet: Removed LNI 192.168.202.143@tcp [ 1551.104147] Key type .llcrypt unregistered [ 1551.109341] Key type ._llcrypt unregistered [ 1573.274784] Key type ._llcrypt registered [ 1573.278794] Key type .llcrypt registered [ 1573.414267] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1574.578796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1574.751085] alg: No test for adler32 (adler32-zlib) [ 1575.820637] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 1576.087878] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1577.767314] Key type lgssc registered [ 1578.764925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1585.338070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1585.354778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.807753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.413355] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1596.390598] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1596.404637] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1599.697215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.072108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1603.959732] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1611.222578] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1617.475447] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1625.093478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.410138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.137736] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1631.541954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1637.690776] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1647.381945] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1651.198675] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1663.417424] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 0h55m16s [ 1664.674924] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1666.542380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1666.548327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.563598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.619388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.633438] Lustre: Skipped 2 previous similar messages [ 1670.907606] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.43@tcp (stopping) [ 1672.434348] Lustre: server umount lustre-MDT0000 complete [ 1673.699636] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1676.306193] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1678.060316] LustreError: 52188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.090083] LustreError: 52188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1682.470685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.608416] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1682.966900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1683.031914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1684.005149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1687.388178] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1691.098484] Lustre: DEBUG MARKER: Wait for df (36746) ... [ 1692.630713] Lustre: DEBUG MARKER: done [ 1698.274074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1698.278695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.288137] Lustre: Skipped 2 previous similar messages [ 1698.295270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1698.298280] Lustre: Skipped 1 previous similar message [ 1699.308200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.776556] Lustre: server umount lustre-OST0000 complete [ 1709.536332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1709.540502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.552338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.570578] Lustre: Skipped 1 previous similar message [ 1709.580611] Lustre: Skipped 3 previous similar messages [ 1712.727470] Lustre: server umount lustre-MDT0000 complete [ 1716.119697] LustreError: 50383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374570 with bad export cookie 6700735892807206114 [ 1716.130097] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1716.131547] LustreError: 50383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1716.361181] Lustre: server umount lustre-MDT0001 complete [ 1727.220705] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1730.161767] Key type lgssc unregistered [ 1730.483507] LNet: 55326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.495679] LNetError: 55326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.566568] LNet: Removed LNI 192.168.202.143@tcp [ 1732.289801] Key type .llcrypt unregistered [ 1732.292928] Key type ._llcrypt unregistered [ 1770.215184] Key type ._llcrypt registered [ 1770.217417] Key type .llcrypt registered [ 1770.334426] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 1771.732532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1771.779578] alg: No test for adler32 (adler32-zlib) [ 1772.857370] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 1773.097816] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 1774.760180] Key type lgssc registered [ 1775.799841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1782.708366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1782.723558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.950407] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1783.978816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1784.156272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.484801] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1797.379149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.452848] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1797.493232] Lustre: 57537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1797.748370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1802.383629] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1809.775938] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1815.122706] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1821.877687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1822.043459] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1822.216162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.512412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1828.347566] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 1835.088878] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1841.399365] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 0h58m15s [ 1842.879051] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1850.829572] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1853.625623] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1853.666690] LustreError: 59985:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1874.250134] Lustre: lustre-MDT0000: Client d5c67473-683c-4ecc-b7eb-8ef33c7ac1fb (at 192.168.202.43@tcp) reconnecting [ 1893.743206] LustreError: 59985:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1896.332260] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1898.642532] Lustre: DEBUG MARKER: done [ 1909.728973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.732139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1909.748638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1909.757411] Lustre: Skipped 1 previous similar message [ 1911.157372] Lustre: server umount lustre-OST0000 complete [ 1914.854674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.882039] Lustre: Skipped 1 previous similar message [ 1914.893745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.942812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.946697] Lustre: Skipped 2 previous similar messages [ 1920.998283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.139178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.146912] Lustre: Skipped 2 previous similar messages [ 1929.183151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1929.469822] Lustre: server umount lustre-MDT0000 complete [ 1933.348372] LustreError: 57527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374788 with bad export cookie 8413092201873108078 [ 1933.352251] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.367731] LustreError: 57527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1933.837449] Lustre: server umount lustre-MDT0001 complete [ 1945.352829] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 1948.441163] Key type lgssc unregistered [ 1948.765990] LNet: 61869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.773315] LNetError: 61869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.787223] LNet: Removed LNI 192.168.202.143@tcp [ 1949.851989] Key type .llcrypt unregistered [ 1949.854065] Key type ._llcrypt unregistered [ 1998.883667] Key type ._llcrypt registered [ 1998.885879] Key type .llcrypt registered [ 1998.995671] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2000.051744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2000.309516] alg: No test for adler32 (adler32-zlib) [ 2001.478469] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 2001.888065] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2003.511411] Key type lgssc registered [ 2004.867188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2008.960892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2008.983658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.297270] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2010.335316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2010.551096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.716369] Lustre: Failing over lustre-MDT0000 [ 2012.029318] Lustre: server umount lustre-MDT0000 complete [ 2018.788147] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:01:12 (1787374872) [ 2025.662728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.977906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.779601] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2038.540985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.658290] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2038.688314] Lustre: 64681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2038.884623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.071615] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2050.832372] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2056.468401] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2063.658237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2063.916103] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2064.153254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.238275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2071.398579] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2079.171921] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2085.731822] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2095.072438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2095.079702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.092562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.134526] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.137621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.160070] Lustre: Skipped 1 previous similar message [ 2097.175283] Lustre: Skipped 2 previous similar messages [ 2100.103757] Lustre: server umount lustre-MDT0000 complete [ 2104.542276] LustreError: 64674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374959 with bad export cookie 11804203877543275281 [ 2104.560664] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.971645] Lustre: server umount lustre-MDT0001 complete [ 2108.703384] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2114.087248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2123.487559] Lustre: 63592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374962/real 1787374962] req@00000000a64b44f4 x1874198377047680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787374978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2123.504906] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.942561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.794387] Lustre: 63593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374964/real 1787374964] req@00000000148ec125 x1874198377047808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787374980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.863239] Lustre: 63594:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374967/real 1787374967] req@0000000042753c4e x1874198377047936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787374983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.202939] LustreError: 66527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.217058] LustreError: 66527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2130.289508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.445797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2131.298886] Lustre: 63591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374969/real 1787374969] req@0000000020236a5e x1874198377048192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787374985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2132.329133] LustreError: 66528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.361294] LustreError: 66527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.395940] LustreError: 66527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2135.394380] Lustre: 63591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374974/real 1787374974] req@000000002b3fa681 x1874198377048320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787374990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2135.481824] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2138.594128] LustreError: 68449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.626428] LustreError: 68449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2143.716209] LustreError: 68450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.744588] LustreError: 68450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2146.067563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.518822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2151.249568] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2151.913420] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2152.943772] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2159.750904] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2165.892090] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2173.411339] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.421157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.423239] Lustre: Skipped 1 previous similar message [ 2178.531068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.542475] Lustre: Skipped 4 previous similar messages [ 2181.643182] Lustre: server umount lustre-MDT0000 complete [ 2183.654307] LustreError: 66528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.675475] LustreError: 66528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2185.232504] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375039 with bad export cookie 11804203877543277556 [ 2185.234668] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.247531] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2185.501207] Lustre: server umount lustre-MDT0001 complete [ 2191.124865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2200.635733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.087179] Lustre: 63592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375043/real 1787375043] req@0000000036648584 x1874198377077248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787375059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2205.109847] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.116820] Lustre: Skipped 1 previous similar message [ 2210.272434] LustreError: 63590:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000010ebdd4f x1874198377078144/t0(0) o250->MGC192.168.202.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2210.541629] LustreError: 69183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.651114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.712944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2213.742150] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2215.670583] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2224.268717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.721595] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2229.737072] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2235.934992] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2241.809340] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2250.211453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.220234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2250.246043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.263979] Lustre: Skipped 2 previous similar messages [ 2254.600969] Lustre: server umount lustre-MDT0000 complete [ 2255.332540] LustreError: 71134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.349460] LustreError: 71134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2258.338063] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375113 with bad export cookie 11804203877543278935 [ 2258.344330] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.354046] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2258.699960] Lustre: server umount lustre-MDT0001 complete [ 2265.308817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2269.223659] Lustre: DEBUG MARKER: files compared the same [ 2281.462455] Lustre: server umount lustre-OST0000 complete [ 2297.588396] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2300.348095] Key type lgssc unregistered [ 2300.668763] LNet: 74412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.677662] LNetError: 74412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.688231] LNet: Removed LNI 192.168.202.143@tcp [ 2301.379744] Key type .llcrypt unregistered [ 2301.381891] Key type ._llcrypt unregistered [ 2328.658525] Key type ._llcrypt registered [ 2328.661883] Key type .llcrypt registered [ 2328.777519] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2329.952835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2330.029689] alg: No test for adler32 (adler32-zlib) [ 2330.970712] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 2331.232166] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2332.871710] Key type lgssc registered [ 2334.192257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2342.942363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2342.971224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.543259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.421377] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug +malloc all [ 2355.167643] LustreError: 75830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.184078] LustreError: 75830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2358.059850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.379765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2363.045485] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug +malloc all [ 2369.800277] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2374.884931] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2381.166655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2381.424558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.933366] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug +malloc all [ 2389.644782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2393.873754] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2404.321296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2404.339063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.356039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.344984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2405.349095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.358154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.374141] Lustre: Skipped 1 previous similar message [ 2408.463764] Lustre: server umount lustre-OST0000 complete [ 2414.565620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2414.575345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.594701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.015175] Lustre: server umount lustre-MDT0000 complete [ 2420.706833] LustreError: 75827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.720941] LustreError: 75827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2422.931159] LustreError: 75810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375277 with bad export cookie 1539683335146848442 [ 2422.953366] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2423.375378] Lustre: server umount lustre-MDT0001 complete [ 2436.403948] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2439.388978] Key type lgssc unregistered [ 2439.678856] LNet: 79368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.693593] LNetError: 79368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.728900] LNet: Removed LNI 192.168.202.143@tcp [ 2440.668216] Key type .llcrypt unregistered [ 2440.669902] Key type ._llcrypt unregistered [ 2465.722550] Key type ._llcrypt registered [ 2465.724456] Key type .llcrypt registered [ 2465.870761] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2467.175194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2467.197134] alg: No test for adler32 (adler32-zlib) [ 2468.353990] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 2468.572102] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2470.279444] Key type lgssc registered [ 2471.765893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2479.872995] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2479.890300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2495.456382] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2500.578229] LustreError: 80763:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [ 2527.263453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.420688] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2540.879559] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2548.372268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.456619] Lustre: *** cfs_fail_loc=706, val=0*** [ 2548.784773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.902439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2553.487664] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2553.830103] LustreError: 80789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.842641] LustreError: 80789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2558.946756] LustreError: 80788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.024584] LustreError: 81849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.051523] LustreError: 81849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2562.577548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.027809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2568.061151] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2575.270891] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2580.983309] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2583.532740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2583.534317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.553457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2583.575050] Lustre: Skipped 1 previous similar message [ 2583.590806] Lustre: Skipped 1 previous similar message [ 2588.647256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.770787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.784199] Lustre: Skipped 2 previous similar messages [ 2597.343156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2597.515247] Lustre: server umount lustre-OST0000 complete [ 2603.012985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.036429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2603.048870] Lustre: Skipped 1 previous similar message [ 2608.096407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.104683] Lustre: Skipped 1 previous similar message [ 2609.413339] Lustre: server umount lustre-MDT0000 complete [ 2613.220229] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.235033] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2613.671754] LustreError: 80770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375468 with bad export cookie 4671380732707960484 [ 2613.679239] LustreError: 80770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2613.682431] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2614.162549] Lustre: server umount lustre-MDT0001 complete [ 2625.151970] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 2628.183452] Key type lgssc unregistered [ 2628.542772] LNet: 84384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.549336] LNetError: 84384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.565191] LNet: Removed LNI 192.168.202.143@tcp [ 2629.586437] Key type .llcrypt unregistered [ 2629.588712] Key type ._llcrypt unregistered [ 2656.190635] Key type ._llcrypt registered [ 2656.192635] Key type .llcrypt registered [ 2656.294607] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2657.233917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2657.415343] alg: No test for adler32 (adler32-zlib) [ 2658.462714] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 2658.676444] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 2660.375367] Key type lgssc registered [ 2661.514370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2668.932553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2668.970738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.557383] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2682.216152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.888525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.008729] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2687.972150] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.982324] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2693.096967] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.114311] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2694.061145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.270239] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.288170] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2694.319761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.207793] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2699.744716] LustreError: 86496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.751452] LustreError: 86496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2704.885857] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.918752] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2707.943218] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2707.964657] Lustre: MGS: Client bb53caba-a94c-4817-a386-d14e6b521940 (at 0@lo) reconnecting [ 2708.007146] LustreError: 86507:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2708.446412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.551559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2713.022389] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2728.929267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2728.937072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.949079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.956626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.957924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.994595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2731.880816] Lustre: server umount lustre-OST0000 complete [ 2740.202137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.209178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2740.210593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2740.210602] Lustre: Skipped 1 previous similar message [ 2740.213161] Lustre: Skipped 1 previous similar message [ 2741.743885] Lustre: server umount lustre-MDT0000 complete [ 2745.313540] LustreError: 86496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2745.341051] LustreError: 86496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2746.230871] Lustre: server umount lustre-MDT0001 complete [ 2749.838110] Lustre: server umount MGS complete [ 2760.030392] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:13:33 (1787375613) [ 2787.977059] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 2798.474934] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 2836.302759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.401309] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2852.028639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.290915] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2852.354280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2852.443813] Lustre: lustre-MDT0001: new disk, initializing [ 2852.564281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2852.619330] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2852.630233] Lustre: 93658:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2852.642124] LustreError: 93658:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2857.991977] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2869.875286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2870.118805] Lustre: lustre-OST0000: new disk, initializing [ 2870.123726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2870.143208] Lustre: 94425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2870.238536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.532177] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2879.584667] Lustre: 94881:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000074adeb95 x1874199065960448/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787375750 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2885.672741] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2892.185608] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2892.194925] Lustre: MGS: Client cdcccd21-099a-4bab-a870-432e6059cacf (at 0@lo) reconnecting [ 2892.233310] Lustre: 92926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2892.253708] LustreError: 93644:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2892.266850] LustreError: 93644:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2892.288174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2892.349823] Lustre: lustre-MDT0000: new disk, initializing [ 2892.444224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.460862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2892.537291] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2893.536720] Lustre: 95493:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000009ca6c33 x1874199065966976/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787375764 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2896.970929] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 2899.936155] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2902.643516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2902.671994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2902.746267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2927.595997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2927.597330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.605571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2927.605583] Lustre: Skipped 1 previous similar message [ 2927.642224] Lustre: Skipped 1 previous similar message [ 2932.709362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2932.718550] Lustre: Skipped 1 previous similar message [ 2932.956479] Lustre: server umount lustre-OST0000 complete [ 2937.825336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.841785] Lustre: Skipped 1 previous similar message [ 2942.944817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.950356] Lustre: Skipped 4 previous similar messages [ 2943.453608] Lustre: server umount lustre-MDT0000 complete [ 2961.375618] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2961.624858] Lustre: server umount lustre-MDT0001 complete [ 2965.843938] Lustre: server umount MGS complete [ 2977.601638] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:17:11 (1787375831) [ 2999.546969] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 3003.294578] Key type lgssc unregistered [ 3003.563680] LNet: 97997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3003.570368] LNetError: 97997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3003.588923] LNet: Removed LNI 192.168.202.143@tcp [ 3004.381507] Key type .llcrypt unregistered [ 3004.387833] Key type ._llcrypt unregistered [ 3025.349127] Key type ._llcrypt registered [ 3025.352360] Key type .llcrypt registered [ 3025.560846] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 3027.134539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3027.235918] alg: No test for adler32 (adler32-zlib) [ 3028.366618] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 3028.606580] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 3030.343182] Key type lgssc registered [ 3031.758353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3041.056284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3041.093476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.846615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.111369] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3053.025808] LustreError: 99221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.051641] LustreError: 99221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3057.127470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.601022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3063.070181] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3071.531169] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3078.129604] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3085.580279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3085.884496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.933566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3093.184194] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3102.972207] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3112.933361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.953931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3112.975174] Lustre: Skipped 1 previous similar message [ 3112.993602] Lustre: Skipped 1 previous similar message [ 3117.634534] Lustre: server umount lustre-OST0000 complete [ 3123.169475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.180684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.288127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.290555] Lustre: Skipped 1 previous similar message [ 3129.081992] Lustre: server umount lustre-MDT0000 complete [ 3132.385302] LustreError: 99221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.410493] LustreError: 99221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3133.455649] LustreError: 99203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375988 with bad export cookie 11587917129075196502 [ 3133.462568] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3133.467882] LustreError: 99203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3133.888813] Lustre: server umount lustre-MDT0001 complete [ 3145.465682] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 3148.433869] Key type lgssc unregistered [ 3148.750750] LNet: 102760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.761047] LNetError: 102760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3148.776423] LNet: Removed LNI 192.168.202.143@tcp [ 3149.617504] Key type .llcrypt unregistered [ 3149.627211] Key type ._llcrypt unregistered [ 3168.298305] Key type ._llcrypt registered [ 3168.301603] Key type .llcrypt registered [ 3168.501629] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 3170.181047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3170.250234] alg: No test for adler32 (adler32-zlib) [ 3171.673027] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 3172.119295] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 3173.959183] Key type lgssc registered [ 3175.613124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3182.490592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3182.497804] LustreError: 103829:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3182.618425] LustreError: 103844:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3182.633257] LustreError: 103829:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4891 [ 3182.649179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.650047] LustreError: 103829:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3182.650067] LustreError: 103829:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3184.142252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3193.009570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.312314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3193.490886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3198.408470] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3204.605936] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing lsmod [ 3207.492686] LustreError: 105047:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3207.529804] LustreError: 105054:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3207.535658] LustreError: 105047:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4965 [ 3207.545746] LustreError: 105047:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3207.550718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3207.552651] LustreError: 105047:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3207.950407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.026896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3214.306446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3214.307865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.314485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3214.360931] Lustre: Skipped 2 previous similar messages [ 3217.427574] Lustre: server umount lustre-MDT0000 complete [ 3218.467784] LustreError: 103877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.488829] LustreError: 103877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3221.389845] LustreError: 103863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376076 with bad export cookie 7635593877220256521 [ 3221.400637] LustreError: 103863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3221.401654] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3222.623375] LustreError: 104314:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002efb751e x1874199603859840/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3222.866406] Lustre: server umount lustre-MDT0001 complete [ 3237.535794] Lustre: server umount lustre-OST0000 complete [ 3247.654167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.127260] LustreError: 106118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3248.223028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.218788] LustreError: 106119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.095107] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3257.322443] LustreError: 106118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.432431] LustreError: 106119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.368244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.972619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3270.666696] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3279.362799] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3285.559365] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3293.966669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3294.487565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3301.319632] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3310.288850] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3318.016401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3328.481847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.485408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3328.513231] Lustre: Skipped 1 previous similar message [ 3328.544400] Lustre: Skipped 3 previous similar messages [ 3332.819730] Lustre: server umount lustre-OST0000 complete [ 3336.688111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3336.704598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.722289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3338.727572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.064797] Lustre: server umount lustre-MDT0000 complete [ 3343.843142] LustreError: 106118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.868187] LustreError: 106118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3348.282506] LustreError: 106100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376202 with bad export cookie 7635593877220257459 [ 3348.299483] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3349.016666] Lustre: server umount lustre-MDT0001 complete [ 3364.338553] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 3368.275366] Key type lgssc unregistered [ 3368.596964] LNet: 109678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3368.607618] LNetError: 109678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3369.638785] LNet: Removed LNI 192.168.202.143@tcp [ 3370.282124] Key type .llcrypt unregistered [ 3370.284394] Key type ._llcrypt unregistered [ 3396.367429] Key type ._llcrypt registered [ 3396.371657] Key type .llcrypt registered [ 3396.530770] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 3398.180673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3398.327200] alg: No test for adler32 (adler32-zlib) [ 3399.500952] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 3399.785931] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 3401.527169] Key type lgssc registered [ 3402.860740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3411.018628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3411.053119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.502555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.481091] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3422.690731] LustreError: 111096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.723702] LustreError: 111096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3427.813673] LustreError: 111097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.315686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3428.833405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3434.915978] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3445.465290] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3453.448596] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3462.287856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3462.713935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.330215] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3479.511622] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3483.203481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3489.365771] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3493.960610] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3498.465070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3498.477423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.483194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3498.486926] Lustre: Skipped 1 previous similar message [ 3498.495362] Lustre: Skipped 1 previous similar message [ 3503.591219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.604132] Lustre: Skipped 1 previous similar message [ 3504.458993] Lustre: server umount lustre-OST0000 complete [ 3508.706457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.709099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3508.731608] Lustre: Skipped 1 previous similar message [ 3508.741870] Lustre: Skipped 1 previous similar message [ 3513.831412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.854572] Lustre: Skipped 1 previous similar message [ 3514.915710] Lustre: server umount lustre-MDT0000 complete [ 3518.949378] LustreError: 111091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.982994] LustreError: 111091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3520.135175] LustreError: 111077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376374 with bad export cookie 12792448430110106101 [ 3520.147719] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.155174] LustreError: 111077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3520.712239] Lustre: server umount lustre-MDT0001 complete [ 3536.627520] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 3540.222958] Key type lgssc unregistered [ 3540.672749] LNet: 114742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3540.681026] LNetError: 114742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3540.713320] LNet: Removed LNI 192.168.202.143@tcp [ 3541.600468] Key type .llcrypt unregistered [ 3541.602614] Key type ._llcrypt unregistered [ 3561.905624] Key type ._llcrypt registered [ 3561.907843] Key type .llcrypt registered [ 3562.073672] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 3563.561691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3563.799129] alg: No test for adler32 (adler32-zlib) [ 3565.029488] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 3565.355807] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 3567.159245] Key type lgssc registered [ 3568.483475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3577.194740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3577.227669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.891500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.199380] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3589.099914] LustreError: 115964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.139400] LustreError: 115964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3594.209253] LustreError: 115965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.530247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.942512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3600.710141] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3609.725693] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3618.468557] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3627.085470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.724627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.248751] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3645.048751] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3649.243059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3659.748069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.771030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3659.781598] Lustre: Skipped 1 previous similar message [ 3664.464784] Lustre: server umount lustre-OST0000 complete [ 3669.987863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3670.004583] Lustre: Skipped 1 previous similar message [ 3670.011720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3670.018125] Lustre: Skipped 1 previous similar message [ 3674.943373] Lustre: server umount lustre-MDT0000 complete [ 3675.109933] LustreError: 115959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.126139] LustreError: 115959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3679.508816] LustreError: 115945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376534 with bad export cookie 12746032999508981653 [ 3679.521462] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3679.531263] LustreError: 115945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3679.957467] Lustre: server umount lustre-MDT0001 complete [ 3691.943536] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 3694.520307] Key type lgssc unregistered [ 3694.879443] LNet: 119527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3694.891691] LNetError: 119527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3694.911862] LNet: Removed LNI 192.168.202.143@tcp [ 3695.753354] Key type .llcrypt unregistered [ 3695.755448] Key type ._llcrypt unregistered [ 3719.963496] Key type ._llcrypt registered [ 3719.966603] Key type .llcrypt registered [ 3720.089350] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 3721.420887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3721.586980] alg: No test for adler32 (adler32-zlib) [ 3722.650364] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 3722.866556] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 3724.567617] Key type lgssc registered [ 3725.849604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3734.364370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3734.387393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.857583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.212704] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3746.293294] LustreError: 120944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3746.323190] LustreError: 120944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3749.209832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.845722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3755.459360] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3764.215979] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3772.440805] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3780.134040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3780.567259] LustreError: 122765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3780.858917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.231259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3787.262113] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3794.053869] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3801.413617] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3805.335626] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3805.343985] Lustre: Skipped 2 previous similar messages [ 3809.431268] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3809.505977] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3810.263112] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3810.267825] Lustre: Skipped 2 previous similar messages [ 3818.614306] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3818.632157] Lustre: Skipped 1 previous similar message [ 3824.232247] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.43@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3824.237533] Lustre: Skipped 2 previous similar messages [ 3826.377810] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.43@tcp 192.168.0.[2,10]@tcp [ 3839.614927] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.43/24@tcp in log lustre-MDT0000 [ 3839.635206] Lustre: Skipped 2 previous similar messages [ 3847.260519] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.43/24@tcp [ 3847.264313] Lustre: Skipped 1 previous similar message [ 3852.780913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3852.791690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.795440] LustreError: Skipped 1 previous similar message [ 3852.839595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.792224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.795334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.959400] Lustre: server umount lustre-OST0000 complete [ 3862.502952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3862.519262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3862.535791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3862.548135] Lustre: Skipped 1 previous similar message [ 3869.152508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3869.158772] Lustre: Skipped 3 previous similar messages [ 3874.273692] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3874.279064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3874.292954] Lustre: Skipped 1 previous similar message [ 3874.562592] Lustre: server umount lustre-MDT0000 complete [ 3879.400208] LustreError: 120944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3879.422223] LustreError: 120944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3879.812369] LustreError: 120924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376734 with bad export cookie 8541973893853740803 [ 3879.813618] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3879.830076] LustreError: 120924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3880.375543] Lustre: server umount lustre-MDT0001 complete [ 3895.549749] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 3898.533608] Key type lgssc unregistered [ 3898.820194] LNet: 125904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3898.847731] LNetError: 125904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3899.887551] LNet: Removed LNI 192.168.202.143@tcp [ 3900.951113] Key type .llcrypt unregistered [ 3900.952682] Key type ._llcrypt unregistered [ 3934.520589] Key type ._llcrypt registered [ 3934.523483] Key type .llcrypt registered [ 3934.669101] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 3935.959643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3936.006490] alg: No test for adler32 (adler32-zlib) [ 3937.413284] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 3937.822746] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 3939.767197] Key type lgssc registered [ 3942.154072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3952.187421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3952.220333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3953.769580] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3953.803799] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.43/24@tcp [ 3953.829741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3958.991955] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3963.368931] LustreError: 127471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.392119] LustreError: 127471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3968.501311] LustreError: 127472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.598647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3968.988723] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3969.041519] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.43/24@tcp [ 3969.123890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3974.689301] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 3983.075332] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3989.597603] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3996.078253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3996.355514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3998.384342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4003.767695] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4012.704869] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4024.295704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.308271] Lustre: Skipped 1 previous similar message [ 4024.310453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.326262] Lustre: Skipped 1 previous similar message [ 4028.933695] Lustre: server umount lustre-OST0000 complete [ 4034.529944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4034.545940] Lustre: Skipped 1 previous similar message [ 4034.549352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4034.553454] Lustre: Skipped 1 previous similar message [ 4038.860572] Lustre: server umount lustre-MDT0000 complete [ 4039.650905] LustreError: 128218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.673973] LustreError: 128218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4042.866814] LustreError: 127453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376897 with bad export cookie 2223848325106207978 [ 4042.868769] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4042.877550] LustreError: 127453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4043.178800] Lustre: server umount lustre-MDT0001 complete [ 4054.209882] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4057.037201] Key type lgssc unregistered [ 4057.270655] LNet: 131064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4057.279262] LNetError: 131064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4057.298647] LNet: Removed LNI 192.168.202.143@tcp [ 4058.201216] Key type .llcrypt unregistered [ 4058.203340] Key type ._llcrypt unregistered [ 4082.756799] Key type ._llcrypt registered [ 4082.759500] Key type .llcrypt registered [ 4082.864805] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 4097.212988] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4098.301989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4098.364725] alg: No test for adler32 (adler32-zlib) [ 4099.474169] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 4099.785616] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4101.527186] Key type lgssc registered [ 4102.722458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4139.391527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4139.441234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.812465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4140.889132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4141.047651] Lustre: lustre-MDT0000: new disk, initializing [ 4141.166531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4141.193623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4145.848943] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4157.340485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.555952] Lustre: 134467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4157.630156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4157.633401] Lustre: Skipped 1 previous similar message [ 4157.763045] Lustre: lustre-MDT0001: new disk, initializing [ 4157.901581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4157.938979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4157.961783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4162.471448] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4172.420954] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4178.421244] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4185.146917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4185.443425] Lustre: lustre-OST0000: new disk, initializing [ 4185.446824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4185.456601] Lustre: 136435:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4185.569488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4191.708414] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4195.411351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4195.415746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4195.467840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4200.452681] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4205.315306] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4205.532871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4210.626477] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4210.802733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4216.288867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4216.289832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.299069] LustreError: Skipped 1 previous similar message [ 4216.302859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4216.313710] Lustre: Skipped 1 previous similar message [ 4216.331905] Lustre: Skipped 1 previous similar message [ 4219.097566] Lustre: server umount lustre-OST0000 complete [ 4224.481220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4224.488863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.508674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4226.529666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4226.530499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.490626] Lustre: server umount lustre-MDT0000 complete [ 4233.635774] LustreError: 135261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377088 with bad export cookie 2561603121057015409 [ 4233.640929] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4233.642696] LustreError: 135261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4233.939436] Lustre: server umount lustre-MDT0001 complete [ 4244.977308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.903276] LustreError: 138652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.924321] LustreError: 138652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4246.033231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.109056] LustreError: 138653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.124074] LustreError: 138653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4251.919851] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4256.236742] LustreError: 138652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4261.354255] LustreError: 138653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.258605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4263.787677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4269.120909] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4277.185892] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4283.413353] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4290.432739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.536462] LustreError: 140472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.875253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4297.826390] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4306.476334] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4312.389784] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4312.659584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4317.547904] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4317.869684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4327.222328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.447740] Lustre: lustre-OST0001: new disk, initializing [ 4327.466976] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4327.483158] Lustre: 142025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4327.599436] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4333.650742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4333.662366] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4333.834434] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4334.391939] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4343.530538] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4343.758502] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4349.061906] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4349.327986] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4356.522741] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b2542beb000.ost_server_uuid 50 [ 4358.996709] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b2542beb000.ost_server_uuid in IDLE state after 0 sec [ 4369.903552] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.909306] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4369.924777] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4369.930654] LustreError: Skipped 1 previous similar message [ 4369.949626] Lustre: Skipped 2 previous similar messages [ 4375.021152] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4375.049390] Lustre: Skipped 1 previous similar message [ 4376.059903] Lustre: server umount lustre-OST0001 complete [ 4380.140047] LustreError: 141191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4380.174502] LustreError: 141191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4383.204840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4383.207187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.212286] Lustre: Skipped 1 previous similar message [ 4383.215874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4386.907817] Lustre: server umount lustre-OST0000 complete [ 4391.904618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4391.917650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.933310] Lustre: Skipped 1 previous similar message [ 4397.788688] Lustre: server umount lustre-MDT0000 complete [ 4400.608794] LustreError: 141817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.629840] LustreError: 141817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4401.912321] LustreError: 138635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377256 with bad export cookie 2561603121057016291 [ 4401.918509] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4401.919772] LustreError: 138635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4402.307834] Lustre: server umount lustre-MDT0001 complete [ 4424.142780] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4427.384622] Key type lgssc unregistered [ 4427.664981] LNet: 145034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4427.674577] LNetError: 145034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4427.691516] LNet: Removed LNI 192.168.202.143@tcp [ 4428.550502] Key type .llcrypt unregistered [ 4428.552863] Key type ._llcrypt unregistered [ 4471.806308] Key type ._llcrypt registered [ 4471.808824] Key type .llcrypt registered [ 4471.968868] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 4487.805610] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4488.964821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4489.089180] alg: No test for adler32 (adler32-zlib) [ 4490.183939] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 4490.422118] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4492.087890] Key type lgssc registered [ 4493.217336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4530.016647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4530.067238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.324166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4531.351378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4531.415985] Lustre: lustre-MDT0000: new disk, initializing [ 4531.468334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4531.477826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4536.880190] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4548.770490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4548.931197] Lustre: 149193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4548.996367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4549.001439] Lustre: Skipped 1 previous similar message [ 4549.090195] Lustre: lustre-MDT0001: new disk, initializing [ 4549.197872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4549.248777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4549.269284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4554.582848] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4565.683900] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4572.825750] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4581.209785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4581.652373] Lustre: lustre-OST0000: new disk, initializing [ 4581.658766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4581.678906] Lustre: 151164:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4581.844551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.639828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4588.658710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4588.686830] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4588.846087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4599.034362] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4604.886819] Lustre: Failing over lustre-OST0000 [ 4605.050378] Lustre: server umount lustre-OST0000 complete [ 4622.815103] Lustre: 147416:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377461/real 1787377461] req@000000000d9ff378 x1874200986947328/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787377477 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4622.832253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.296924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.697426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.752234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4625.788069] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4626.720230] Lustre: 147416:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377464/real 1787377464] req@0000000043ee1c04 x1874200986947456/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787377480 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4627.769860] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4630.815827] Lustre: 147416:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377469/real 1787377469] req@00000000eee2a944 x1874200986949376/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787377485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4630.836379] Lustre: 147416:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4633.372049] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4633.860618] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4633.863402] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4635.936039] Lustre: 147417:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377474/real 1787377474] req@00000000313efc24 x1874200986950400/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787377490 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4635.974557] Lustre: 147417:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4637.914135] Lustre: Failing over lustre-MDT0000 [ 4638.182165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.198939] Lustre: Skipped 1 previous similar message [ 4638.332424] Lustre: server umount lustre-MDT0000 complete [ 4643.302175] LustreError: 150221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.320911] LustreError: 150221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4644.076572] LustreError: 149200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.417557] LustreError: 149204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.434478] LustreError: 149204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4653.412178] Lustre: 147417:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377492/real 1787377492] req@0000000041c55c14 x1874200986960384/t0(0) o400->MGC192.168.202.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787377508 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4653.438012] Lustre: 147417:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4653.453366] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4653.473519] LustreError: 149205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4653.485898] LustreError: 149205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4658.661108] LustreError: 149199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.681956] LustreError: 149199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4659.243219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.794310] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6d429d426195483d [ 4663.812183] Lustre: MGC192.168.202.143@tcp: Connection restored to 0@lo (at 0@lo) [ 4663.818134] Lustre: Skipped 1 previous similar message [ 4664.240753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.342813] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4665.483947] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4669.435424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4669.463214] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4669.556323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4670.452141] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4680.161101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4680.168146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.170772] LustreError: Skipped 1 previous similar message [ 4680.180854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.188730] Lustre: Skipped 2 previous similar messages [ 4680.205415] Lustre: Skipped 1 previous similar message [ 4684.778360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4684.793313] Lustre: Skipped 1 previous similar message [ 4686.391343] Lustre: server umount lustre-OST0000 complete [ 4695.009522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4695.012540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.028383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4695.051527] Lustre: Skipped 1 previous similar message [ 4695.070568] Lustre: Skipped 1 previous similar message [ 4696.975622] Lustre: server umount lustre-MDT0000 complete [ 4700.129775] LustreError: 149201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.156657] LustreError: 149201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4700.897777] LustreError: 149988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377555 with bad export cookie 7873028007027623997 [ 4700.899915] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4700.929283] LustreError: 149988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4701.387571] Lustre: server umount lustre-MDT0001 complete [ 4712.505432] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 4716.644753] Key type lgssc unregistered [ 4717.073950] LNet: 154819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4717.088086] LNetError: 154819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4718.129672] LNet: Removed LNI 192.168.202.143@tcp [ 4719.489237] Key type .llcrypt unregistered [ 4719.490956] Key type ._llcrypt unregistered [ 4758.374721] Key type ._llcrypt registered [ 4758.379276] Key type .llcrypt registered [ 4758.504215] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4759.879495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4759.915603] alg: No test for adler32 (adler32-zlib) [ 4761.016393] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 4761.320262] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 4762.983224] Key type lgssc registered [ 4764.404763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4773.448138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4773.468114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.133236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4780.622929] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4784.609233] LustreError: 156342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.627408] LustreError: 156342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4789.731249] LustreError: 156343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.117470] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4794.850730] LustreError: 156342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4795.519320] LustreError: 156371:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4795.527530] LustreError: 156371:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4795.536926] LustreError: 156371:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4796.068820] Lustre: server umount lustre-MDT0000 complete [ 4813.835762] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 4824.475985] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 4864.113428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.656442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4864.734255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4864.890150] Lustre: lustre-MDT0000: new disk, initializing [ 4865.005219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4865.060101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4870.564594] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4882.872952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.045321] Lustre: 159675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4883.129619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4883.136307] Lustre: Skipped 1 previous similar message [ 4883.239888] Lustre: lustre-MDT0001: new disk, initializing [ 4883.391701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4883.451123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4883.470115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4889.001617] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4898.476474] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4905.638476] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4913.214883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.476793] Lustre: lustre-OST0000: new disk, initializing [ 4913.479890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4913.485548] Lustre: 161643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4913.576115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4920.357670] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 4923.493989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4923.513928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4923.650490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4931.129392] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4936.027659] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4938.495770] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:49:51 (1787377791) [ 5200.354434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.368239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5200.371419] Lustre: Skipped 2 previous similar messages [ 5200.380331] Lustre: Skipped 2 previous similar messages [ 5204.875293] Lustre: server umount lustre-MDT0000 complete [ 5205.475623] LustreError: 159682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5205.500955] LustreError: 159682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5208.932891] LustreError: 160471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378063 with bad export cookie 13382963913021615096 [ 5208.939438] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5208.946951] LustreError: 160471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5210.592313] LustreError: 159681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5210.592620] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.654314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5213.600722] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5218.722102] LustreError: 159681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5218.723220] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5218.740116] LustreError: 159681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5223.391170] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5223.724454] Lustre: server umount lustre-MDT0001 complete [ 5242.335297] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5242.629191] Lustre: server umount lustre-OST0000 complete [ 5254.164659] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 5265.961701] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5304.023182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.364748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5304.414308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5304.540691] Lustre: lustre-MDT0000: new disk, initializing [ 5304.675501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.700802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5309.829272] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5321.271625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.403649] Lustre: 166316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5321.457135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5321.464233] Lustre: Skipped 1 previous similar message [ 5321.588939] Lustre: lustre-MDT0001: new disk, initializing [ 5321.686082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5321.731225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5321.749718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5327.151068] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5337.946735] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5344.251615] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5352.318226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5352.626329] Lustre: lustre-OST0000: new disk, initializing [ 5352.629603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5352.635469] Lustre: 168284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5352.724166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5358.197797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5358.205705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5358.398368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5360.500068] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5371.921343] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5377.508516] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5377.721774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5382.592654] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5382.858941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5389.283900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5389.287478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.302087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5391.163482] Lustre: server umount lustre-OST0000 complete [ 5398.504148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5398.508963] LustreError: Skipped 1 previous similar message [ 5398.516318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.536890] Lustre: Skipped 1 previous similar message [ 5398.551962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5398.559720] Lustre: Skipped 1 previous similar message [ 5401.496984] Lustre: server umount lustre-MDT0000 complete [ 5404.642944] LustreError: 166323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.691195] LustreError: 166323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5405.998388] LustreError: 167110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378260 with bad export cookie 13382963913021773926 [ 5406.019283] LustreError: 167110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5406.041444] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5406.501752] Lustre: server umount lustre-MDT0001 complete [ 5415.328170] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:57:48 (1787378268) [ 5434.782754] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 5445.938296] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5487.028336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.416284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5487.461326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5487.548392] Lustre: lustre-MDT0000: new disk, initializing [ 5487.637210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.663204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5493.113438] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5504.449849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5504.642463] Lustre: 173042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5504.999955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5510.036613] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5520.171147] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5526.730723] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5534.486022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5534.744853] Lustre: lustre-OST0000: new disk, initializing [ 5534.747148] Lustre: Skipped 1 previous similar message [ 5534.755323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5534.761580] Lustre: Skipped 2 previous similar messages [ 5534.765926] Lustre: 175020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5542.145668] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5542.582579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5542.596029] Lustre: Skipped 1 previous similar message [ 5542.605431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5542.657211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5552.977987] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5563.376359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5563.385001] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5563.393627] Lustre: Skipped 1 previous similar message [ 5563.405990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5563.412435] LustreError: Skipped 1 previous similar message [ 5563.414673] Lustre: Skipped 2 previous similar messages [ 5568.495598] LustreError: 175401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5568.842925] Lustre: server umount lustre-OST0000 complete [ 5573.602401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.611673] Lustre: Skipped 2 previous similar messages [ 5579.968475] Lustre: server umount lustre-MDT0000 complete [ 5584.385412] LustreError: 173036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378439 with bad export cookie 13382963913021774808 [ 5584.400389] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5584.406942] LustreError: 173036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5585.331840] Lustre: server umount lustre-MDT0001 complete [ 5595.881846] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:00:49 (1787378449) [ 5615.929532] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 5625.812815] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5670.595329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5670.912957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5670.938579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5671.039154] Lustre: lustre-MDT0000: new disk, initializing [ 5671.195768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5671.198652] Lustre: Skipped 2 previous similar messages [ 5671.234476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5676.663179] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5688.311599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.540879] Lustre: 179360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5688.899171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5695.041506] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5706.237738] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5711.799415] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5719.911537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5720.369641] Lustre: 181337:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5728.744942] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5729.884191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5730.064096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5740.813941] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5750.754836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5750.770526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5750.790966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5750.793675] Lustre: Skipped 5 previous similar messages [ 5754.473508] Lustre: server umount lustre-OST0000 complete [ 5760.488870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5760.498628] LustreError: Skipped 1 previous similar message [ 5765.548939] Lustre: server umount lustre-MDT0000 complete [ 5766.113322] LustreError: 181169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.136899] LustreError: 181169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5769.822787] LustreError: 179350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378624 with bad export cookie 13382963913021775893 [ 5769.823911] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5769.838516] LustreError: 179350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5770.270647] Lustre: server umount lustre-MDT0001 complete [ 5783.088970] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 5786.644096] Key type lgssc unregistered [ 5786.991934] LNet: 183197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5787.011614] LNetError: 183197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5787.037731] LNet: Removed LNI 192.168.202.143@tcp [ 5788.134514] Key type .llcrypt unregistered [ 5788.137833] Key type ._llcrypt unregistered [ 5819.057853] Key type ._llcrypt registered [ 5819.059596] Key type .llcrypt registered [ 5819.189757] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5820.587451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5820.600227] alg: No test for adler32 (adler32-zlib) [ 5821.663547] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 5821.960901] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 5823.619122] Key type lgssc registered [ 5824.902721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5834.698819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5834.719204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5836.186760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.544689] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5845.474115] LustreError: 184616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.490554] LustreError: 184616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5850.596863] LustreError: 184615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.983667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.374291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5858.376426] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5866.506845] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5872.980510] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5879.616919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5880.008836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.020386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5886.781278] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 5894.710677] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5917.152433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5917.162310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5917.175691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.176159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5918.178107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.183128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5918.214772] Lustre: Skipped 1 previous similar message [ 5922.523885] Lustre: server umount lustre-OST0000 complete [ 5928.421571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.426448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5928.438720] Lustre: Skipped 1 previous similar message [ 5928.464481] Lustre: Skipped 1 previous similar message [ 5932.696042] Lustre: server umount lustre-MDT0000 complete [ 5933.541858] LustreError: 184608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.579912] LustreError: 184608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5936.689445] LustreError: 184595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378791 with bad export cookie 8735255422817697578 [ 5936.692825] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5936.701198] LustreError: 184595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5937.412943] Lustre: server umount lustre-MDT0001 complete [ 5949.564618] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 5952.878523] Key type lgssc unregistered [ 5953.155794] LNet: 188180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5953.161978] LNetError: 188180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5953.190353] LNet: Removed LNI 192.168.202.143@tcp [ 5954.074885] Key type .llcrypt unregistered [ 5954.077229] Key type ._llcrypt unregistered [ 5978.100619] Key type ._llcrypt registered [ 5978.103221] Key type .llcrypt registered [ 5978.223898] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 5979.162313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5979.253343] alg: No test for adler32 (adler32-zlib) [ 5980.530522] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 5980.861780] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 5982.623972] Key type lgssc registered [ 5984.095753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5992.048516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5992.085368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5993.626248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.003824] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6004.194521] LustreError: 189598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.215451] LustreError: 189598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6008.967755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6009.372252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6013.976839] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6021.775881] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6027.549587] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6034.893053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6035.442620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.528987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6041.989439] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6050.598786] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6063.358720] Lustre: lustre-OST0000: Not available for connect from 192.168.202.43@tcp (stopping) [ 6064.099534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.103391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6064.135406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6064.147414] Lustre: Skipped 1 previous similar message [ 6068.471148] Lustre: lustre-OST0000: Not available for connect from 192.168.202.43@tcp (stopping) [ 6073.584276] Lustre: lustre-OST0000: Not available for connect from 192.168.202.43@tcp (stopping) [ 6073.595070] Lustre: Skipped 2 previous similar messages [ 6074.847178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6075.045789] Lustre: server umount lustre-OST0000 complete [ 6083.632349] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b254c56e800.ost_server_uuid 50 [ 6085.543167] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6101.473407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6101.488950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.506890] Lustre: Skipped 1 previous similar message [ 6101.515913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6101.525422] Lustre: Skipped 2 previous similar messages [ 6104.624673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.296327] Lustre: server umount lustre-MDT0000 complete [ 6109.672717] LustreError: 189599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.709263] LustreError: 189599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6111.634696] LustreError: 189578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378966 with bad export cookie 6111975736677033014 [ 6111.639617] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6112.002299] Lustre: server umount lustre-MDT0001 complete [ 6122.927628] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:09:35 (1787378975) [ 6130.224721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6130.824351] LustreError: 193514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.909270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6136.214590] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6136.292268] LustreError: 193515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6140.397601] LustreError: 193514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6146.470128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.820623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6152.260403] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6160.079089] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6166.174374] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6173.393537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.886017] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.952507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6180.991956] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6188.740507] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6196.357697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6196.619614] Lustre: lustre-OST0001: new disk, initializing [ 6196.623118] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6196.628431] Lustre: 196359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6196.730701] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6203.574190] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6206.555381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6206.565600] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6206.696324] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6213.845257] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6217.202282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6217.212899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.217609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.227188] Lustre: Skipped 1 previous similar message [ 6217.244671] Lustre: Skipped 3 previous similar messages [ 6222.884429] Lustre: server umount lustre-OST0000 complete [ 6225.143841] LustreError: 195338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6225.163722] LustreError: 195338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6230.118374] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6230.385185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6235.675394] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6235.894672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6242.791348] LustreError: 195336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.802829] LustreError: 195336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6253.028682] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6253.033991] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6253.042743] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6253.042753] Lustre: Skipped 3 previous similar messages [ 6253.046866] Lustre: Skipped 1 previous similar message [ 6258.198306] Lustre: server umount lustre-OST0001 complete [ 6263.264919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.284237] Lustre: Skipped 1 previous similar message [ 6269.745619] Lustre: server umount lustre-MDT0000 complete [ 6275.591403] LustreError: 193494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379130 with bad export cookie 6111975736677034092 [ 6275.608927] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6275.616819] LustreError: 193494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6275.975519] Lustre: server umount lustre-MDT0001 complete [ 6305.233683] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:12:39 (1787379159) [ 6312.980420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.594989] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6313.670423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6313.938785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6319.369736] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6329.447270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.521665] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6329.556165] Lustre: 199799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6335.543576] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6344.119470] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6351.116646] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6359.538562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6359.742462] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6366.395558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6367.133733] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6376.889752] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6385.551743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6385.671734] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6386.088642] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6386.101424] Lustre: Skipped 2 previous similar messages [ 6392.172510] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6394.960377] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6403.886075] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6407.136691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6407.143880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.170593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6407.190268] Lustre: Skipped 5 previous similar messages [ 6412.769039] LustreError: 201653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.785757] LustreError: 201653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6412.832466] Lustre: server umount lustre-OST0000 complete [ 6433.251914] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6433.271330] LustreError: Skipped 1 previous similar message [ 6435.873258] Lustre: server umount lustre-OST0001 complete [ 6442.469747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6442.486060] Lustre: Skipped 3 previous similar messages [ 6446.056858] Lustre: server umount lustre-MDT0000 complete [ 6449.951690] LustreError: 200548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379304 with bad export cookie 6111975736677035499 [ 6449.974952] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6450.492747] Lustre: server umount lustre-MDT0001 complete [ 6475.881588] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:15:29 (1787379329) [ 6492.955202] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 6505.616051] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 6546.697342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.998840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6547.066627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6547.160307] Lustre: lustre-MDT0000: new disk, initializing [ 6547.248733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.266592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6552.070902] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6562.655508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6562.846572] Lustre: 207907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6562.959103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6562.967197] Lustre: Skipped 1 previous similar message [ 6563.038989] Lustre: lustre-MDT0001: new disk, initializing [ 6563.160422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6563.165224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6568.023361] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6577.979799] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6584.668709] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6592.443695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6592.926909] Lustre: lustre-OST0000: new disk, initializing [ 6592.930843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6592.937910] Lustre: 209874:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6600.253916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6600.266748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6600.397785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6600.416180] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6613.522114] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6618.035785] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6618.419912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6622.908248] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6623.075934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6626.277079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6626.282666] LustreError: Skipped 1 previous similar message [ 6626.284079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.290031] Lustre: Skipped 1 previous similar message [ 6626.295348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6626.301564] Lustre: Skipped 8 previous similar messages [ 6631.448668] LustreError: 209855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6631.487965] LustreError: 209855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6631.713672] Lustre: server umount lustre-OST0000 complete [ 6642.800675] Lustre: server umount lustre-MDT0000 complete [ 6647.236536] LustreError: 207899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379501 with bad export cookie 6111975736677037284 [ 6647.237780] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6647.250427] LustreError: 207899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6648.153575] Lustre: server umount lustre-MDT0001 complete [ 6660.329468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6668.106289] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6677.679543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6682.838638] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6693.904835] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6701.278954] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6708.537673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6716.897627] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6726.056190] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6731.571367] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6731.860892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6737.968624] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6738.308873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6741.984232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6742.007057] LustreError: Skipped 1 previous similar message [ 6747.007608] Lustre: server umount lustre-OST0000 complete [ 6755.215824] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6760.818938] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6790.829470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6798.640931] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6806.515182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6806.534965] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6806.539042] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6806.569448] Lustre: Skipped 1 previous similar message [ 6807.426978] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6809.483320] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6816.744750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6816.753677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6816.754793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6816.754804] Lustre: Skipped 6 previous similar messages [ 6816.765180] Lustre: Skipped 6 previous similar messages [ 6820.778509] Lustre: server umount lustre-OST0000 complete [ 6840.287381] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6840.535578] Lustre: server umount lustre-MDT0000 complete [ 6842.343166] LustreError: 212823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.375923] LustreError: 212823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6845.093799] LustreError: 212073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379699 with bad export cookie 6111975736677038166 [ 6845.115196] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6845.724438] Lustre: server umount lustre-MDT0001 complete [ 6856.506283] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:21:49 (1787379709) [ 6865.077715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.654335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6865.676672] Lustre: Skipped 6 previous similar messages [ 6871.371400] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6881.712549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6888.123472] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6896.534272] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6902.348553] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6909.914474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6917.371845] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6925.418630] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6931.248942] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6931.489217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6936.369319] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6936.583775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6943.504695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6943.670512] Lustre: lustre-OST0001: new disk, initializing [ 6943.674315] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6943.680224] Lustre: 221435:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6948.989055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6948.998280] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6949.123543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6950.609526] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 6960.609806] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6965.921918] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6966.205914] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6971.655967] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6971.945259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6974.946033] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6974.967820] LustreError: Skipped 2 previous similar messages [ 6988.255179] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6988.510261] Lustre: server umount lustre-OST0001 complete [ 6995.916115] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6996.128155] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7001.051963] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7001.268303] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7031.795456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7037.413816] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7037.432550] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7037.442594] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7037.479560] Lustre: Skipped 1 previous similar message [ 7039.704416] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7050.202462] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7052.138299] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7061.046540] Lustre: server umount lustre-OST0001 complete [ 7080.420466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7080.438150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7080.459697] Lustre: Skipped 7 previous similar messages [ 7080.483439] Lustre: Skipped 16 previous similar messages [ 7086.509474] Lustre: server umount lustre-OST0000 complete [ 7097.957571] Lustre: server umount lustre-MDT0000 complete [ 7098.853027] LustreError: 218793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.865811] LustreError: 218793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 7102.498762] LustreError: 221434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379957 with bad export cookie 6111975736677039132 [ 7102.511650] LustreError: 221434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7102.517586] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7109.038360] Lustre: server umount lustre-MDT0001 complete [ 7139.000164] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:26:32 (1787379992) [ 7148.081620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.497410] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7148.552532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7153.758709] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7163.976369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7164.164577] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7164.214607] Lustre: 226680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7170.270351] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7179.207317] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7186.062477] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7193.558110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7193.790988] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7201.953713] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7210.896528] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7222.791631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7223.012974] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7230.717226] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7239.146788] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7241.245798] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7247.031381] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7247.326818] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7252.862614] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7253.230140] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7260.348434] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b25446e2000.ost_server_uuid 50 [ 7262.395462] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b25446e2000.ost_server_uuid in IDLE state after 0 sec [ 7263.663832] Lustre: Permanently deactivating lustre-OST0001 [ 7263.679806] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7268.445901] Lustre: Permanently reactivating lustre-OST0001 [ 7268.484847] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7268.492873] Lustre: Skipped 2 previous similar messages [ 7273.450788] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.470554] LustreError: Skipped 3 previous similar messages [ 7279.144929] Lustre: server umount lustre-OST0001 complete [ 7289.982229] Lustre: server umount lustre-OST0000 complete [ 7300.992260] Lustre: server umount lustre-MDT0000 complete [ 7305.743030] LustreError: 226671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380160 with bad export cookie 6111975736677040420 [ 7305.755202] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.174693] Lustre: server umount lustre-MDT0001 complete [ 7333.075908] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:29:46 (1787380186) [ 7342.050656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7342.324651] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7342.378837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7342.398353] Lustre: Skipped 2 previous similar messages [ 7348.205747] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7360.079833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7360.325069] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7360.397769] Lustre: 233381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7366.268349] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7375.642274] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7383.688908] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7391.987148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7392.294557] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7392.362967] Lustre: Permanently deactivating lustre-OST0000 [ 7392.373987] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7392.580044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7392.582794] Lustre: Skipped 10 previous similar messages [ 7398.986917] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7400.544938] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7410.096113] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7418.881487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7419.017108] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7427.043397] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7427.228200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7436.055335] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7438.220136] Lustre: Permanently reactivating lustre-OST0000 [ 7438.254392] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7438.266626] Lustre: Skipped 2 previous similar messages [ 7447.107929] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7447.111712] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 48 seconds. I think it's dead, and I am evicting it. exp 000000007179842c, cur 1787380301 deadline 1787380283 last 1787380253 [ 7447.126380] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7447.149207] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7447.207338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7463.105395] Lustre: server umount lustre-OST0001 complete [ 7481.311135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7481.572984] Lustre: server umount lustre-OST0000 complete [ 7491.962126] Lustre: server umount lustre-MDT0000 complete [ 7495.948475] LustreError: 233373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380350 with bad export cookie 6111975736677042093 [ 7495.954945] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7495.964353] LustreError: 233373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7496.376557] Lustre: server umount lustre-MDT0001 complete [ 7508.763298] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 7512.819416] Key type lgssc unregistered [ 7513.352945] LNet: 238269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7513.371538] LNetError: 238269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7513.390771] LNet: Removed LNI 192.168.202.143@tcp [ 7514.283561] Key type .llcrypt unregistered [ 7514.287201] Key type ._llcrypt unregistered [ 7541.694563] Key type ._llcrypt registered [ 7541.704845] Key type .llcrypt registered [ 7541.789223] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 7542.827397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7542.840804] alg: No test for adler32 (adler32-zlib) [ 7543.915473] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 7544.156358] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 7545.807618] Key type lgssc registered [ 7547.230506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7558.545880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7558.575127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.054313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7565.374210] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7569.383175] LustreError: 239733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7569.395868] LustreError: 239733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7574.246974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.429705] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7574.491505] Lustre: 239723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7574.511063] Lustre: Permanently deactivating lustre-MDT0001 [ 7574.522981] LustreError: 239734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7574.571902] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7574.956284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7581.221292] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7584.895282] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7591.383112] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7599.949661] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7609.071690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7609.444626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.051562] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7619.686922] LustreError: 241576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7619.699638] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7625.193118] LustreError: 241576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7626.102923] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7630.314891] LustreError: 241574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.351058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7630.356098] LustreError: 241574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7635.853333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7636.133213] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7641.590581] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7641.601945] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7643.335447] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7651.114590] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7654.893840] Lustre: Permanently reactivating lustre-MDT0001 [ 7654.924369] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7654.936117] Lustre: Skipped 2 previous similar messages [ 7661.327966] Lustre: lustre-MDT0001: Client 9e806d3c-3069-4a7b-a866-bf733d64a5f2 (at 192.168.202.43@tcp) reconnecting [ 7661.687675] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7661.710515] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7661.720902] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7661.727452] LustreError: 243302:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7661.727905] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7661.735988] LustreError: 243302:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7661.736125] LustreError: 243302:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.143@tcp: cfg command failed: rc = -108 [ 7661.736169] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7661.736169] [ 7661.736338] LustreError: 239726:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7668.875484] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7670.480611] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7676.900316] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7677.132740] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7682.893733] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7683.103046] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7686.346767] Lustre: Permanently deactivating lustre-MDT0001 [ 7686.369262] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7686.374274] Lustre: Skipped 2 previous similar messages [ 7692.974298] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7702.480262] Lustre: server umount lustre-MDT0000 complete [ 7703.009931] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.030145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7703.056769] LustreError: 239728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7703.097788] LustreError: 239728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7707.241752] LustreError: 239715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380561 with bad export cookie 160592664379869677 [ 7707.248521] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7707.259571] LustreError: 239715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7708.029854] Lustre: server umount lustre-MDT0001 complete [ 7714.929335] Lustre: server umount lustre-OST0000 complete [ 7728.757127] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:36:22 (1787380582) [ 7741.544913] Lustre: server umount lustre-OST0001 complete [ 7749.446698] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 7761.216980] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 7802.949500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.294836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7803.303730] Lustre: Skipped 2 previous similar messages [ 7803.332700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7803.404634] Lustre: lustre-MDT0000: new disk, initializing [ 7803.473784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7803.496651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7809.122207] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7821.952519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7822.233148] Lustre: 247998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7822.321398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7822.330778] Lustre: Skipped 1 previous similar message [ 7822.432971] Lustre: lustre-MDT0001: new disk, initializing [ 7822.557215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7822.606810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7822.632406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7827.561384] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7839.047847] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7846.178591] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7854.624621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.985340] Lustre: lustre-OST0000: new disk, initializing [ 7854.988181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7854.998142] Lustre: 249970:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7855.119742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.841632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7860.873631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7861.064466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7863.051846] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7874.321346] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7877.331316] LustreError: 248006:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7887.428560] LustreError: 248006:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7887.460754] LustreError: 249108:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7889.790189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7890.138754] Lustre: lustre-OST0001: new disk, initializing [ 7890.145193] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7890.156209] Lustre: 251084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7890.315804] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7896.756936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7896.776944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7896.905118] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7897.519282] LustreError: 249108:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7898.830894] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 7909.924290] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7911.716936] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7917.547670] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7917.555368] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.564526] Lustre: Skipped 4 previous similar messages [ 7917.571316] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7917.578401] Lustre: Skipped 1 previous similar message [ 7922.671223] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7922.679481] Lustre: Skipped 1 previous similar message [ 7927.788348] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7927.804939] Lustre: Skipped 1 previous similar message [ 7928.287609] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7928.746420] Lustre: server umount lustre-OST0001 complete [ 7932.903159] LustreError: 249955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7932.946953] LustreError: 249955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7953.399235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7953.401031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.404909] LustreError: Skipped 1 previous similar message [ 7953.407643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7953.420844] Lustre: Skipped 1 previous similar message [ 7955.825328] Lustre: server umount lustre-OST0000 complete [ 7963.621409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.656783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7963.670852] Lustre: Skipped 2 previous similar messages [ 7968.169168] Lustre: server umount lustre-MDT0000 complete [ 7968.738012] LustreError: 251390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.779891] LustreError: 251390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7972.386173] LustreError: 247992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380827 with bad export cookie 160592664379872491 [ 7972.391857] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7972.399964] LustreError: 247992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7978.745163] Lustre: server umount lustre-MDT0001 complete [ 7994.895473] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 7999.033228] Key type lgssc unregistered [ 7999.356786] LNet: 253122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7999.367941] LNetError: 253122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7999.389995] LNet: Removed LNI 192.168.202.143@tcp [ 8000.581754] Key type .llcrypt unregistered [ 8000.587221] Key type ._llcrypt unregistered [ 8050.122780] Key type ._llcrypt registered [ 8050.123891] Key type .llcrypt registered [ 8050.192691] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 8051.295418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8051.461742] alg: No test for adler32 (adler32-zlib) [ 8052.762471] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 8053.120401] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 8054.743238] Key type lgssc registered [ 8055.927650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8066.075453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8066.117844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.507525] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8067.544260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8067.847431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.011580] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8084.756201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8084.976702] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8085.052535] Lustre: 255334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8085.307329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8091.141342] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8100.343784] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8108.300346] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8116.692943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8116.931074] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8117.070221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8124.528130] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8127.109626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8134.203431] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8193.505345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8193.513172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8193.533514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8194.043590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8194.046233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8194.064581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.099201] Lustre: Skipped 1 previous similar message [ 8194.584460] Lustre: server umount lustre-OST0000 complete [ 8201.771867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8221.382955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.962850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8230.039108] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8234.223630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8234.263177] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8239.789767] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8243.544576] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8243.573913] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8279.525810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8279.527093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8279.534200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8279.555622] Lustre: Skipped 1 previous similar message [ 8284.640535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8284.647817] Lustre: Skipped 1 previous similar message [ 8285.005822] Lustre: server umount lustre-OST0000 complete [ 8289.764183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.789119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8289.797241] Lustre: Skipped 2 previous similar messages [ 8300.001212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.015167] Lustre: Skipped 3 previous similar messages [ 8303.071210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8303.524482] Lustre: server umount lustre-MDT0000 complete [ 8307.768299] LustreError: 255328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381162 with bad export cookie 18213065968947124589 [ 8307.774365] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8307.791379] LustreError: 255328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8308.402825] Lustre: server umount lustre-MDT0001 complete [ 8321.943638] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 8326.459065] Key type lgssc unregistered [ 8326.866410] LNet: 260766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8326.872125] LNetError: 260766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8326.886650] LNet: Removed LNI 192.168.202.143@tcp [ 8327.844565] Key type .llcrypt unregistered [ 8327.849458] Key type ._llcrypt unregistered [ 8358.135755] Key type ._llcrypt registered [ 8358.145488] Key type .llcrypt registered [ 8358.329257] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 8359.993043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8360.080799] alg: No test for adler32 (adler32-zlib) [ 8361.139410] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 8361.461123] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 8363.215608] Key type lgssc registered [ 8364.694470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8374.088663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8374.113439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.812086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8381.849429] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8386.021190] LustreError: 262184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8386.029716] LustreError: 262184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8391.140414] LustreError: 262183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8391.796351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8392.085490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8397.705749] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8407.852458] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8416.289981] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8425.537684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8426.150151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8434.162580] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8444.377426] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8445.628771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8481.761269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8481.770312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8481.782065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8481.814594] Lustre: Skipped 1 previous similar message [ 8485.228402] Lustre: server umount lustre-OST0000 complete [ 8492.006284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.015134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8492.017087] Lustre: Skipped 1 previous similar message [ 8492.035746] Lustre: Skipped 2 previous similar messages [ 8496.559308] Lustre: server umount lustre-MDT0000 complete [ 8497.120929] LustreError: 264629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8497.142400] LustreError: 264629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8501.444705] LustreError: 262163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381356 with bad export cookie 5763542972367622449 [ 8501.446803] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8501.463479] LustreError: 262163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8501.909069] Lustre: server umount lustre-MDT0001 complete [ 8517.399505] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 8521.194551] Key type lgssc unregistered [ 8521.588500] LNet: 266502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8521.596588] LNetError: 266502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8522.662788] LNet: Removed LNI 192.168.202.143@tcp [ 8524.073402] Key type .llcrypt unregistered [ 8524.075916] Key type ._llcrypt unregistered [ 8547.038796] Key type ._llcrypt registered [ 8547.044993] Key type .llcrypt registered [ 8547.190592] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 8548.741854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8548.990114] alg: No test for adler32 (adler32-zlib) [ 8550.163901] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 8550.430805] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 8552.207241] Key type lgssc registered [ 8553.547617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8564.029570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8564.043970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8565.577704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8572.033327] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8575.980836] LustreError: 267724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8575.998786] LustreError: 267724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8581.098973] LustreError: 267725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8582.848398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.348061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8589.519261] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8599.617409] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8606.656719] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8615.102454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8615.663348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8624.460604] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8633.567448] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8637.199661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8652.777178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.779374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8652.818143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8652.842094] Lustre: Skipped 1 previous similar message [ 8653.290207] Lustre: server umount lustre-OST0000 complete [ 8657.889057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8657.894333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8657.913570] Lustre: Skipped 2 previous similar messages [ 8657.941204] Lustre: Skipped 1 previous similar message [ 8663.008414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8663.020353] Lustre: Skipped 1 previous similar message [ 8668.128962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8668.144243] Lustre: Skipped 1 previous similar message [ 8672.227669] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8672.385515] Lustre: server umount lustre-MDT0000 complete [ 8673.253145] LustreError: 267720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8673.277464] LustreError: 267720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8677.372376] LustreError: 267706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381532 with bad export cookie 10654777321272451563 [ 8677.376337] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8677.389656] LustreError: 267706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8677.848262] Lustre: server umount lustre-MDT0001 complete [ 8691.225293] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 8695.026023] Key type lgssc unregistered [ 8695.293150] LNet: 271406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8695.298795] LNetError: 271406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8696.358926] LNet: Removed LNI 192.168.202.143@tcp [ 8697.422728] Key type .llcrypt unregistered [ 8697.425527] Key type ._llcrypt unregistered [ 8723.828569] Key type ._llcrypt registered [ 8723.834545] Key type .llcrypt registered [ 8724.039625] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 8725.546538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8725.581415] alg: No test for adler32 (adler32-zlib) [ 8726.685435] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 8726.953883] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 8728.681381] Key type lgssc registered [ 8729.991589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8738.780709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8738.821586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8740.304429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8746.356315] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8749.537675] LustreError: 272827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.569062] LustreError: 272827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8754.658714] LustreError: 272826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8757.924299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8758.389980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8763.617968] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8773.944322] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8780.671614] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8789.121649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8789.396779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8795.796345] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8804.810526] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8809.905874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8834.530158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8834.539737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8834.554501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8835.553731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8835.560616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.562415] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8839.106643] Lustre: server umount lustre-OST0000 complete [ 8845.280594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8845.285485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.294259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8845.297542] Lustre: Skipped 1 previous similar message [ 8849.934143] Lustre: server umount lustre-MDT0000 complete [ 8850.921413] LustreError: 272821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.930854] LustreError: 272821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8854.280090] LustreError: 272806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381708 with bad export cookie 11209930972595043725 [ 8854.294543] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8854.295085] LustreError: 272806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8854.573415] Lustre: server umount lustre-MDT0001 complete [ 8867.857285] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 8871.544331] Key type lgssc unregistered [ 8871.871846] LNet: 277139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8871.910707] LNetError: 277139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8871.949023] LNet: Removed LNI 192.168.202.143@tcp [ 8873.015249] Key type .llcrypt unregistered [ 8873.017395] Key type ._llcrypt unregistered [ 8891.545671] Key type ._llcrypt registered [ 8891.551393] Key type .llcrypt registered [ 8891.652692] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 8892.790787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8892.909909] alg: No test for adler32 (adler32-zlib) [ 8894.532663] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 8895.014711] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 8896.799170] Key type lgssc registered [ 8898.766691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8908.067733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8908.102194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8909.700193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8914.560110] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8920.035850] LustreError: 278373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8920.060245] LustreError: 278373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8922.175614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.509349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8926.515743] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8934.689288] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8940.148807] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8946.558246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8946.895866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8953.262612] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 8955.070122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8961.890240] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8975.848055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8975.883908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8975.886526] Lustre: Skipped 1 previous similar message [ 8975.891823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8975.899783] Lustre: Skipped 1 previous similar message [ 8980.254610] Lustre: server umount lustre-OST0000 complete [ 8986.081479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8986.090331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.103263] Lustre: Skipped 1 previous similar message [ 8986.123401] Lustre: Skipped 1 previous similar message [ 8990.954662] Lustre: server umount lustre-MDT0000 complete [ 8991.202422] LustreError: 278373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8991.243439] LustreError: 278373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8995.102147] LustreError: 278341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381849 with bad export cookie 10291407339732467124 [ 8995.112689] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8995.545027] Lustre: server umount lustre-MDT0001 complete [ 9010.039594] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 9013.360177] Key type lgssc unregistered [ 9013.688630] LNet: 282077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9013.698173] LNetError: 282077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9013.732660] LNet: Removed LNI 192.168.202.143@tcp [ 9014.709565] Key type .llcrypt unregistered [ 9014.716164] Key type ._llcrypt unregistered [ 9027.327423] Key type ._llcrypt registered [ 9027.334754] Key type .llcrypt registered [ 9027.493611] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9047.109497] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 9061.548480] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 9062.801042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9062.817933] alg: No test for adler32 (adler32-zlib) [ 9063.864655] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 9064.173026] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 9065.871204] Key type lgssc registered [ 9067.341476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9102.931554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9102.957488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9104.296986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9104.354145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9104.497447] Lustre: lustre-MDT0000: new disk, initializing [ 9104.598861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9104.633783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9109.372507] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9120.777989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9120.899823] Lustre: 285631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9120.984782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9120.988087] Lustre: Skipped 1 previous similar message [ 9121.067408] Lustre: lustre-MDT0001: new disk, initializing [ 9121.153748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9121.193152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9121.217701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9125.915357] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9135.390311] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9141.763392] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9149.811716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9150.207436] Lustre: lustre-OST0000: new disk, initializing [ 9150.211041] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9150.216955] Lustre: 287601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9150.371344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9157.166412] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9158.741264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9158.754878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9158.879810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9167.531932] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9171.833565] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9172.066376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9176.346185] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9176.535401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9179.626535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9179.633159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9179.639746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9179.659786] LustreError: Skipped 1 previous similar message [ 9184.989273] Lustre: server umount lustre-OST0000 complete [ 9189.860079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9189.865098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9189.897480] Lustre: Skipped 2 previous similar messages [ 9189.918554] Lustre: Skipped 2 previous similar messages [ 9194.978992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9194.988752] Lustre: Skipped 1 previous similar message [ 9195.644240] Lustre: server umount lustre-MDT0000 complete [ 9199.291591] LustreError: 286438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382053 with bad export cookie 3585904405123050551 [ 9199.302403] LustreError: 286438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9199.305062] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9199.617233] Lustre: server umount lustre-MDT0001 complete [ 9208.270391] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:01:01 (1787382061) [ 9215.123328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9215.747576] LustreError: 290024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.773473] LustreError: 290024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9215.857851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9221.057903] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9221.092577] LustreError: 290025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9226.215739] LustreError: 290024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9229.723986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9230.141089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9235.011805] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9243.398739] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9250.511178] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9257.653724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9257.874503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9264.917610] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9272.444555] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9286.627105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9286.642752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9286.668694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9290.722937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9290.724321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9290.732308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9292.598116] Lustre: server umount lustre-OST0000 complete [ 9296.865282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9296.876153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9300.961223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9300.973029] Lustre: Skipped 3 previous similar messages [ 9302.814893] Lustre: server umount lustre-MDT0000 complete [ 9306.082581] LustreError: 290025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9306.115906] LustreError: 290025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9307.194883] LustreError: 290005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382161 with bad export cookie 3585904405123051433 [ 9307.198556] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9307.204482] LustreError: 290005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9307.718666] Lustre: server umount lustre-MDT0001 complete [ 9320.522858] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 9323.690328] Key type lgssc unregistered [ 9324.008719] LNet: 293584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9324.012293] LNetError: 293584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9325.032659] LNet: Removed LNI 192.168.202.143@tcp [ 9326.037551] Key type .llcrypt unregistered [ 9326.039293] Key type ._llcrypt unregistered [ 9354.961973] Key type ._llcrypt registered [ 9354.964756] Key type .llcrypt registered [ 9355.063526] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 9370.075784] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 9371.276209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9371.338572] alg: No test for adler32 (adler32-zlib) [ 9372.513571] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 9372.836487] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 9374.527207] Key type lgssc registered [ 9375.596283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9427.389029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9427.417728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9428.687351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9428.732228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9428.785708] Lustre: lustre-MDT0000: new disk, initializing [ 9428.906095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9428.939519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9434.486069] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9446.937110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.117433] Lustre: 297414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9447.192246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9447.196455] Lustre: Skipped 1 previous similar message [ 9447.273518] Lustre: lustre-MDT0001: new disk, initializing [ 9447.478482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9447.536152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9447.552821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9453.581832] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9463.712682] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9469.721457] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9476.571493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9476.914469] Lustre: lustre-OST2710: new disk, initializing [ 9476.918915] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9476.925465] Lustre: 299382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9477.013118] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9483.318742] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9483.347562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9483.356829] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9483.470812] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9493.091592] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9499.767229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9500.075212] Lustre: lustre-OST03e8: new disk, initializing [ 9500.081485] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9500.091669] Lustre: 300446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9500.233432] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9507.087638] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9508.987207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9508.994888] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9509.086943] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9515.887075] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9524.135418] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9524.390525] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9528.976940] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9529.234611] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9533.704774] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9533.911773] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9538.555626] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9538.730289] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9544.673346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9544.681508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9544.690889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9545.188250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9545.208360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9545.218299] Lustre: Skipped 1 previous similar message [ 9545.233032] Lustre: Skipped 3 previous similar messages [ 9548.978831] Lustre: server umount lustre-MDT0000 complete [ 9553.027357] LustreError: 298207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382407 with bad export cookie 6001902537149902797 [ 9553.048621] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9553.054561] LustreError: 298207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9553.524554] Lustre: server umount lustre-MDT0001 complete [ 9567.588215] Lustre: server umount lustre-OST2710 complete [ 9580.767938] Lustre: server umount lustre-OST03e8 complete [ 9600.346942] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [ 9609.766479] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 9652.720414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9653.045090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9653.077499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9653.198749] Lustre: lustre-MDT0000: new disk, initializing [ 9653.293590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.329423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9658.882967] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9670.496120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.625371] Lustre: 305698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9670.665806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9670.669491] Lustre: Skipped 1 previous similar message [ 9670.730637] Lustre: lustre-MDT0001: new disk, initializing [ 9670.816762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9670.857416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9670.874873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9676.521437] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9688.069768] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9694.807369] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9702.918847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9703.212297] Lustre: lustre-OST0000: new disk, initializing [ 9703.215451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9703.221371] Lustre: 307670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9703.334766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9712.129328] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9712.760932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9712.772870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9712.852720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9723.545718] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9729.141166] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9729.439689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9735.432716] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9735.764834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9738.724259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.725902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9738.726457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9738.745406] Lustre: Skipped 2 previous similar messages [ 9743.850828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9743.861706] Lustre: Skipped 2 previous similar messages [ 9744.295176] Lustre: server umount lustre-OST0000 complete [ 9748.960650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.962614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9748.974171] Lustre: Skipped 1 previous similar message [ 9749.000119] Lustre: Skipped 1 previous similar message [ 9754.738112] Lustre: server umount lustre-MDT0000 complete [ 9759.202462] LustreError: 305709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9759.236424] LustreError: 305709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9759.442264] LustreError: 305689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382614 with bad export cookie 6001902537149904477 [ 9759.451074] LustreError: 305689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9759.453054] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9759.915547] Lustre: server umount lustre-MDT0001 complete [ 9769.612843] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:10:23 (1787382623) [ 9771.193693] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9772.822180] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:10:26 (1787382626) [ 9776.701281] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 9807.164163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9807.481890] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9807.494634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9807.649917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9812.900497] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9821.710857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.856525] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9821.881667] Lustre: 311529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9827.773292] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9850.432928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9850.708597] Lustre: Denying initial registration attempt from nid 192.168.202.143@tcp, specified as failover [ 9850.719093] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9850.731893] LustreError: 312984:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9850.740042] LustreError: 312984:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9850.745773] LustreError: 312984:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9850.866798] Lustre: server umount lustre-OST0000 complete [ 9850.868975] LustreError: 312984:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9863.137677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9863.153664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9863.158185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9863.185561] Lustre: Skipped 1 previous similar message [ 9863.200442] Lustre: Skipped 3 previous similar messages [ 9873.375241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9873.692197] Lustre: server umount lustre-MDT0000 complete [ 9878.384563] LustreError: 312272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382733 with bad export cookie 6001902537149905359 [ 9878.385585] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9878.405089] LustreError: 312272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9878.840821] Lustre: server umount lustre-MDT0001 complete [ 9890.655955] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [ 9894.563640] Key type lgssc unregistered [ 9894.951051] LNet: 314108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9894.963599] LNetError: 314108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9896.039894] LNet: Removed LNI 192.168.202.143@tcp [ 9897.077129] Key type .llcrypt unregistered [ 9897.079736] Key type ._llcrypt unregistered [ 9911.166594] Key type ._llcrypt registered [ 9911.172666] Key type .llcrypt registered [ 9911.394925] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 9912.999401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9913.078851] alg: No test for adler32 (adler32-zlib) [ 9914.210847] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [ 9914.480497] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [ 9916.151660] Key type lgssc registered [ 9917.501986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9967.444471] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [ 9979.070890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9979.088853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.533688] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9980.585928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9980.816521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9986.038881] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [ 9995.057191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9995.143668] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9995.166264] Lustre: 316872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9995.465913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10001.472504] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10008.775662] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10015.350331] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10035.253143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10035.658089] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10035.963480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10043.446306] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10052.798489] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10056.161358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10056.166968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10056.182472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10056.687607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10056.691233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10056.697934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10061.827444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10062.039750] Lustre: server umount lustre-OST0000 complete [10066.913249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10066.915809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10066.932519] Lustre: Skipped 1 previous similar message [10066.956929] Lustre: Skipped 2 previous similar messages [10072.034710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10072.046128] Lustre: Skipped 1 previous similar message [10072.780499] Lustre: server umount lustre-MDT0000 complete [10076.965819] LustreError: 316864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382931 with bad export cookie 13297391589634883138 [10076.975290] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10076.976104] LustreError: 316864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10077.311538] Lustre: server umount lustre-MDT0001 complete [10088.956939] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [10091.779692] Key type lgssc unregistered [10092.093089] LNet: 320450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10092.098213] LNetError: 320450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10092.121799] LNet: Removed LNI 192.168.202.143@tcp [10092.800713] Key type .llcrypt unregistered [10092.803717] Key type ._llcrypt unregistered [10120.151494] Key type ._llcrypt registered [10120.152974] Key type .llcrypt registered [10120.267319] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [10121.851059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10121.995704] alg: No test for adler32 (adler32-zlib) [10123.500719] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [10123.902114] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [10125.663281] Key type lgssc registered [10127.219289] Lustre: Echo OBD driver; http://www.lustre.org/ [10137.373790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10137.405496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10139.333367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10144.211404] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10148.850907] LustreError: 321867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10148.858177] LustreError: 321867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10152.497074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10152.836322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10157.807334] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10166.860661] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10173.266553] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10182.885830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10183.260918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10189.845663] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10198.035831] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10208.225891] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.231055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10208.244335] Lustre: Skipped 2 previous similar messages [10208.260518] Lustre: Skipped 2 previous similar messages [10210.062831] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.43@tcp (stopping) [10212.094221] LustreError: 324452:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000005682fa2b) refcount nonzero (2) after lock cleanup; forcing cleanup. [10212.345796] Lustre: server umount lustre-MDT0000 complete [10213.350629] LustreError: 322598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10213.374055] LustreError: 322598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10215.149963] LustreError: 321863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10216.042355] LustreError: 321850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383070 with bad export cookie 3952355608521284489 [10216.055687] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10218.465078] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10218.465498] LustreError: 321862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10218.485198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10221.472651] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10221.477327] Lustre: Skipped 1 previous similar message [10223.030392] Lustre: server umount lustre-MDT0001 complete [10229.757251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10238.458656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10240.671717] LustreError: 325448:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10240.686221] LustreError: 325448:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000401448bb x1874206893578496/t0(0) o250->MGC192.168.202.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787383095 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10240.704311] LustreError: 325448:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10241.338608] LustreError: 323700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10241.349520] LustreError: 323700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10241.394294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10241.443564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10246.514863] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10250.595865] LustreError: 323700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.609281] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10250.622911] LustreError: 323700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10254.865628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10255.236228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10260.405311] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10260.463386] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10268.386515] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10269.784751] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10275.632767] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10277.152986] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10286.065698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.067595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10286.079396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10286.090698] Lustre: Skipped 1 previous similar message [10288.878449] Lustre: server umount lustre-OST0000 complete [10296.289436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10296.292955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10296.296125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10296.303314] Lustre: Skipped 2 previous similar messages [10299.056887] Lustre: server umount lustre-MDT0000 complete [10301.409753] LustreError: 325465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.419092] LustreError: 325465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10302.643817] LustreError: 321850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383157 with bad export cookie 3952355608521291972 [10302.647885] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10302.651269] LustreError: 321850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10302.887416] Lustre: server umount lustre-MDT0001 complete [10313.389631] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [10316.251961] Key type lgssc unregistered [10316.627681] LNet: 328138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10316.632452] LNetError: 328138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10317.671679] LNet: Removed LNI 192.168.202.143@tcp [10318.518442] Key type .llcrypt unregistered [10318.521047] Key type ._llcrypt unregistered [10343.639224] Key type ._llcrypt registered [10343.641143] Key type .llcrypt registered [10343.730501] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [10344.834523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10344.939459] alg: No test for adler32 (adler32-zlib) [10346.032199] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [10346.299806] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [10347.919398] Key type lgssc registered [10349.051934] Lustre: Echo OBD driver; http://www.lustre.org/ [10357.168202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10357.182894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.562179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10363.299669] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10368.993393] LustreError: 329568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10369.018705] LustreError: 329568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10371.763858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.993253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10376.391739] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10385.715202] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10392.160807] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10397.473218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10397.488319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10397.492279] Lustre: Skipped 1 previous similar message [10401.669118] Lustre: server umount lustre-MDT0000 complete [10402.784744] LustreError: 329569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.801691] LustreError: 329569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10405.113917] LustreError: 329549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383259 with bad export cookie 17258713089411616178 [10405.129223] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10405.133490] LustreError: 329549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10405.407402] Lustre: server umount lustre-MDT0001 complete [10419.410866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10419.668370] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10419.708807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10419.969911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10424.531481] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10433.393537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.477446] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10433.502797] Lustre: 332153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10433.749436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10438.368954] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10447.840476] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10454.758083] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10464.381097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.612036] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10464.627914] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10464.641786] LustreError: 334038:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10464.655481] LustreError: 334038:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10464.675551] LustreError: 334038:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10464.768506] Lustre: server umount lustre-OST0000 complete [10464.771251] LustreError: 334038:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10475.626533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10475.831336] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10476.013161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10482.869971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10483.187977] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10491.100552] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10499.870262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10499.999042] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10500.118501] Lustre: lustre-OST0001: new disk, initializing [10500.124131] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10500.149985] Lustre: 335580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10500.281380] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10507.577720] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10509.895406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10509.938534] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10510.149243] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10516.106144] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10520.545827] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10520.563857] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10520.573486] Lustre: Skipped 1 previous similar message [10520.578791] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10520.588580] Lustre: Skipped 1 previous similar message [10525.687355] LustreError: 334503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10525.709436] LustreError: 334503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10525.852874] Lustre: server umount lustre-OST0001 complete [10530.790161] LustreError: 334503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.790968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.835732] Lustre: Skipped 1 previous similar message [10530.839963] LustreError: 334503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10530.854066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10530.871780] Lustre: Skipped 1 previous similar message [10535.872116] Lustre: server umount lustre-OST0000 complete [10541.025364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10541.043099] Lustre: Skipped 2 previous similar messages [10541.058310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10546.146686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10546.156415] Lustre: Skipped 2 previous similar messages [10554.335867] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10554.575698] Lustre: server umount lustre-MDT0000 complete [10556.384803] LustreError: 332160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10556.396612] LustreError: 332160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10557.819792] LustreError: 335579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383412 with bad export cookie 17258713089411616724 [10557.827456] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10557.828530] LustreError: 335579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10558.308294] Lustre: server umount lustre-MDT0001 complete [10571.655271] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [10574.498812] Key type lgssc unregistered [10574.851825] LNet: 337544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10574.860903] LNetError: 337544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10574.878435] LNet: Removed LNI 192.168.202.143@tcp [10575.872260] Key type .llcrypt unregistered [10575.877731] Key type ._llcrypt unregistered [10649.976552] Key type ._llcrypt registered [10649.978803] Key type .llcrypt registered [10650.081345] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [10663.755673] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [10665.196618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10665.212321] alg: No test for adler32 (adler32-zlib) [10666.295739] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [10666.601436] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [10668.359194] Key type lgssc registered [10669.505156] Lustre: Echo OBD driver; http://www.lustre.org/ [10705.026129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10705.086217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10706.433635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10706.462812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10706.569494] Lustre: lustre-MDT0000: new disk, initializing [10706.692272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10706.749475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10712.560806] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10723.590391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.686747] Lustre: 342480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10723.733030] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10723.736188] Lustre: Skipped 1 previous similar message [10723.795323] Lustre: lustre-MDT0001: new disk, initializing [10723.921846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10724.008215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10724.025254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10729.099552] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10738.404551] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10743.997499] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10750.228481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10750.520991] Lustre: lustre-OST0000: new disk, initializing [10750.527167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10750.534511] Lustre: 344449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10750.608517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10757.786073] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10758.224256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10758.234373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10758.374650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10767.460776] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10772.495638] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10772.814696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10778.711610] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10778.995519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10784.225366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10784.234528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10784.237936] LustreError: Skipped 1 previous similar message [10784.239247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10784.246685] Lustre: Skipped 1 previous similar message [10784.289934] Lustre: Skipped 1 previous similar message [10788.083193] Lustre: server umount lustre-OST0000 complete [10795.491050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10795.506356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10795.572981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10799.589445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10799.590110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10804.707312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10804.717973] Lustre: Skipped 1 previous similar message [10809.311172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10809.570706] Lustre: server umount lustre-MDT0000 complete [10812.944384] LustreError: 342474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383667 with bad export cookie 11281468840555615386 [10812.951633] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10812.959775] LustreError: 342474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10813.391554] Lustre: server umount lustre-MDT0001 complete [10823.205470] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:27:56 (1787383676) [10832.131686] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:28:05 (1787383685) [10841.893282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10842.457588] LustreError: 347208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.487498] LustreError: 347208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10842.587329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10847.741149] LustreError: 347209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10848.286665] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10852.838731] LustreError: 347208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10857.422130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10857.684333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10862.073473] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10869.632820] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10876.265597] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10883.553371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10883.935269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10890.723513] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10899.208366] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10911.760786] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10913.883582] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10915.794742] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10917.373213] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10919.460366] Lustre: Failing over lustre-MDT0000 [10919.957734] Lustre: server umount lustre-MDT0000 complete [10922.268091] LustreError: 347205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10922.980545] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10922.997915] Lustre: Skipped 1 previous similar message [10927.337494] LustreError: 347205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.385812] LustreError: 347205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10937.605886] LustreError: 347204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10937.627878] LustreError: 347204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10939.359751] Lustre: 340706:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383777/real 1787383777] req@00000000a2cc38b9 x1874207462990720/t0(0) o400->MGC192.168.202.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787383793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10939.406473] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10940.475345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.600859] LustreError: 340703:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001747687a x1874207462996736/t0(0) o250->MGC192.168.202.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10949.936313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10950.038504] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10953.974755] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10954.064483] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10955.249392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10955.280229] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10955.324112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10963.567077] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10965.406368] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10969.214617] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10971.274740] Lustre: Failing over lustre-MDT0000 [10971.702302] Lustre: server umount lustre-MDT0000 complete [10974.458655] LustreError: 350117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10974.473556] LustreError: 350117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [10975.712816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10975.713253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10975.730214] Lustre: Skipped 2 previous similar messages [10990.653969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10990.732615] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10990.898263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10990.944705] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10995.086966] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [10995.957746] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10996.217924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10996.233401] Lustre: Skipped 2 previous similar messages [10996.264828] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10996.342211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11004.655362] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11006.153231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11016.674141] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11016.678043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11016.695178] Lustre: Skipped 2 previous similar messages [11016.698429] Lustre: Skipped 2 previous similar messages [11018.169938] Lustre: server umount lustre-MDT0000 complete [11021.796467] LustreError: 347208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11021.811302] LustreError: 347208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [11021.836408] LustreError: 347188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383876 with bad export cookie 11281468840555618095 [11021.840714] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11021.859436] LustreError: 347188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11022.214563] Lustre: server umount lustre-MDT0001 complete [11035.546567] Lustre: server umount lustre-OST0000 complete [11053.150266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11058.380087] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11068.434641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11068.773088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11068.776875] Lustre: Skipped 1 previous similar message [11074.312205] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11083.335626] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11089.247822] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11097.084717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11098.621838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11105.023669] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11114.779327] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11124.194562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11124.201676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11124.217418] Lustre: Skipped 1 previous similar message [11124.222436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11124.240631] Lustre: Skipped 2 previous similar messages [11126.409339] Lustre: server umount lustre-OST0000 complete [11144.682735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.692262] Lustre: Skipped 6 previous similar messages [11145.183382] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11145.415898] Lustre: server umount lustre-MDT0000 complete [11149.691396] LustreError: 353550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384004 with bad export cookie 11281468840555618844 [11149.694908] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11149.701087] LustreError: 353550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11150.085549] Lustre: server umount lustre-MDT0001 complete [11161.255601] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [11164.292358] Key type lgssc unregistered [11164.617749] LNet: 357108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11164.626618] LNetError: 357108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11164.639550] LNet: Removed LNI 192.168.202.143@tcp [11165.591620] Key type .llcrypt unregistered [11165.596848] Key type ._llcrypt unregistered [11193.942531] Key type ._llcrypt registered [11193.947672] Key type .llcrypt registered [11194.070157] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [11195.478170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11195.770070] alg: No test for adler32 (adler32-zlib) [11196.891241] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [11197.332437] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [11199.055303] Key type lgssc registered [11200.411516] Lustre: Echo OBD driver; http://www.lustre.org/ [11209.371938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11209.413920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11211.077490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11215.762308] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11216.360426] LustreError: 358623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11216.375347] LustreError: 358623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11220.453436] LustreError: 358624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11224.595209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.902117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11230.227886] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11240.139406] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11247.883846] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11256.048081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11256.428277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11262.847750] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11271.106971] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11281.042498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11293.115646] Lustre: server umount lustre-MDT0000 complete [11296.744280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11296.771164] Lustre: Skipped 1 previous similar message [11296.772939] LustreError: 358620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.785694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11296.818857] LustreError: 358620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11297.560459] LustreError: 358604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384152 with bad export cookie 3430652973157608684 [11297.570854] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11297.574355] LustreError: 358604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11297.852321] Lustre: server umount lustre-MDT0001 complete [11311.735677] Lustre: server umount lustre-OST0000 complete [11326.892354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11327.548344] LustreError: 362359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.679324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11332.313892] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11333.106533] LustreError: 362360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11342.408416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11342.947757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11347.603637] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11355.854520] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11362.279115] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11371.373847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11371.890759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11372.922590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11379.896985] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11388.951497] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11403.756940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11403.766414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11403.776329] Lustre: Skipped 3 previous similar messages [11403.794129] Lustre: Skipped 2 previous similar messages [11408.463449] Lustre: server umount lustre-MDT0000 complete [11408.869384] LustreError: 362360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.894677] LustreError: 362360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11413.256783] LustreError: 365040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384267 with bad export cookie 3430652973157609797 [11413.261229] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11413.265835] LustreError: 365040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11413.984723] Lustre: server umount lustre-MDT0001 complete [11429.030898] Lustre: server umount lustre-OST0000 complete [11446.720413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.266293] LustreError: 366273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.358291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11451.888316] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11461.335745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11461.606954] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11466.757642] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11477.114513] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11483.899769] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11491.318372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11491.853920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11491.861848] Lustre: Skipped 1 previous similar message [11496.939495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11498.058544] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11505.327744] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11512.291235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11512.299901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.310913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11512.314039] Lustre: Skipped 1 previous similar message [11512.329121] Lustre: Skipped 1 previous similar message [11516.177897] Lustre: server umount lustre-OST0000 complete [11522.531854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11522.547368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11522.560293] Lustre: Skipped 1 previous similar message [11527.648121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.651317] Lustre: Skipped 2 previous similar messages [11534.303184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11534.514051] Lustre: server umount lustre-MDT0000 complete [11537.895958] LustreError: 366274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11537.933416] LustreError: 366274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11538.677120] LustreError: 366255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384393 with bad export cookie 3430652973157610819 [11538.682559] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11538.684478] LustreError: 366255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11539.015288] Lustre: server umount lustre-MDT0001 complete [11550.331849] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [11553.294138] Key type lgssc unregistered [11553.575250] LNet: 369812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11553.584430] LNetError: 369812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11553.606860] LNet: Removed LNI 192.168.202.143@tcp [11554.832188] Key type .llcrypt unregistered [11554.833909] Key type ._llcrypt unregistered [11584.616732] Key type ._llcrypt registered [11584.625480] Key type .llcrypt registered [11584.765628] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [11586.343727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11586.472243] alg: No test for adler32 (adler32-zlib) [11587.871247] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [11588.265569] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [11590.081816] Key type lgssc registered [11591.942472] Lustre: Echo OBD driver; http://www.lustre.org/ [11600.678934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11600.694361] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11600.702724] LustreError: 371256:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11600.732819] LustreError: 371256:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11600.741325] LustreError: 371256:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11600.747889] LustreError: 371256:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11600.756851] LustreError: 371256:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11608.411667] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11608.417673] LustreError: 371653:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11608.439486] LustreError: 371653:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11608.444308] LustreError: 371653:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11608.449166] LustreError: 371653:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11608.454959] LustreError: 371653:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11633.457035] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [11638.008597] Key type lgssc unregistered [11638.523537] LNet: 372662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11638.528722] LNetError: 372662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11638.557808] LNet: Removed LNI 192.168.202.143@tcp [11639.836929] Key type .llcrypt unregistered [11639.840161] Key type ._llcrypt unregistered [11665.517617] Key type ._llcrypt registered [11665.521199] Key type .llcrypt registered [11665.667200] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [11681.218926] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [11682.499189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11682.555209] alg: No test for adler32 (adler32-zlib) [11683.695375] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [11684.006367] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [11685.719313] Key type lgssc registered [11686.844439] Lustre: Echo OBD driver; http://www.lustre.org/ [11727.355945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11727.400135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.769316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11728.824851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11728.944298] Lustre: lustre-MDT0000: new disk, initializing [11729.123758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11729.156918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11734.902930] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11747.433990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11747.613528] Lustre: 375822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11747.756315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11747.760589] Lustre: Skipped 1 previous similar message [11747.855309] Lustre: lustre-MDT0001: new disk, initializing [11747.917124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11747.967801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11747.980190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11752.816231] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11767.421693] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11776.094149] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11784.272306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11784.610590] Lustre: lustre-OST0000: new disk, initializing [11784.613634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11784.618626] Lustre: 377790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11784.723348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11790.927639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11790.952800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11791.153368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11792.103646] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11803.785450] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11809.281482] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11809.508915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11814.740286] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11815.051596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11822.051112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.053857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11822.057231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11822.069020] Lustre: Skipped 1 previous similar message [11827.174903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11827.179302] Lustre: Skipped 1 previous similar message [11831.263245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11831.424566] Lustre: server umount lustre-OST0000 complete [11837.409983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11837.413625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11837.428120] Lustre: Skipped 1 previous similar message [11837.465590] Lustre: Skipped 2 previous similar messages [11842.461519] Lustre: server umount lustre-MDT0000 complete [11847.435215] LustreError: 376616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384702 with bad export cookie 11009644284745408020 [11847.436061] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11847.446270] LustreError: 376616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11847.913866] Lustre: server umount lustre-MDT0001 complete [11858.348367] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11860.819670] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:45:13 (1787384713) [11868.635518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11869.238955] LustreError: 380254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11869.261125] LustreError: 380254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11869.338418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11874.788383] LustreError: 380255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.522535] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11878.881330] LustreError: 380254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.002052] LustreError: 380255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11886.178861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11886.740121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11892.958411] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11902.744267] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11909.856073] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11917.926846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11918.312481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11927.585616] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11937.382694] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11944.655116] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11944.874801] Lustre: lustre-OST0001: new disk, initializing [11944.878511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11944.887138] Lustre: 383096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11944.979660] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11952.487392] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [11954.365579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11954.394121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11954.510920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11963.404681] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11970.017642] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11970.041564] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11970.057699] Lustre: Skipped 1 previous similar message [11970.063514] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11970.089382] Lustre: Skipped 1 previous similar message [11975.034222] Lustre: server umount lustre-OST0001 complete [11975.147948] LustreError: 382078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11975.168978] LustreError: 382078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11985.385588] LustreError: 382079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11985.394858] LustreError: 382079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11994.091416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11994.106560] LustreError: Skipped 1 previous similar message [11994.110578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11994.138121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11994.148164] Lustre: Skipped 1 previous similar message [11999.887704] Lustre: server umount lustre-OST0000 complete [12005.864121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12005.871170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12005.901299] Lustre: Skipped 2 previous similar messages [12005.930663] Lustre: Skipped 3 previous similar messages [12012.563631] Lustre: server umount lustre-MDT0000 complete [12016.097089] LustreError: 380983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12016.113441] LustreError: 380983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12019.254815] LustreError: 383095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384873 with bad export cookie 11009644284745408902 [12019.276907] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12019.284200] LustreError: 383095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12026.110805] Lustre: server umount lustre-MDT0001 complete [12041.999464] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [12045.990948] Key type lgssc unregistered [12046.536251] LNet: 385089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12046.559701] LNetError: 385089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12047.591536] LNet: Removed LNI 192.168.202.143@tcp [12048.688905] Key type .llcrypt unregistered [12048.690606] Key type ._llcrypt unregistered [12107.026303] Key type ._llcrypt registered [12107.029103] Key type .llcrypt registered [12107.143492] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [12108.529777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12108.612186] alg: No test for adler32 (adler32-zlib) [12109.751322] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [12110.124381] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [12111.825718] Key type lgssc registered [12113.065373] Lustre: Echo OBD driver; http://www.lustre.org/ [12122.176276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12122.198686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12123.449873] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12123.489338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12123.800279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12129.378414] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12138.398337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12138.546248] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12138.600957] Lustre: 387892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12138.858030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12143.834430] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12153.055546] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12160.201386] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12162.541204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12162.550226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12162.560201] Lustre: Skipped 1 previous similar message [12162.572779] Lustre: Skipped 1 previous similar message [12167.648154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12167.655753] Lustre: Skipped 1 previous similar message [12168.696429] Lustre: server umount lustre-MDT0000 complete [12172.792772] LustreError: 387883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385027 with bad export cookie 10303752529429419804 [12172.805741] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12172.813794] LustreError: 387883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12173.535199] Lustre: server umount lustre-MDT0001 complete [12184.089328] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:50:37 (1787385037) [12191.059812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12191.623935] LustreError: 390404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12191.643556] LustreError: 390404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12191.680984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12196.510394] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12196.832399] LustreError: 390405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.956149] LustreError: 390404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12205.254344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12205.718270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12211.312562] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12219.541217] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12226.061910] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12233.029600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12233.397617] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12233.694197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12240.753194] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12248.948706] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12256.763012] Lustre: Permanently deactivating lustre-OST0000 [12256.787650] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12266.575520] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12268.677665] LustreError: 393441:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12268.684501] LustreError: 393441:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12268.688817] LustreError: 393441:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12273.002903] LustreError: 393490:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12273.012251] LustreError: 393490:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12280.986835] Lustre: server umount lustre-OST0000 complete [12283.906737] LustreError: 393739:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12283.913957] LustreError: 393739:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12287.459176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12287.471366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12287.484561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12289.013065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12294.117059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12294.124394] Lustre: Skipped 3 previous similar messages [12299.236682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12299.248159] Lustre: Skipped 1 previous similar message [12300.255102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12300.486875] Lustre: server umount lustre-MDT0000 complete [12304.342867] LustreError: 393097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385159 with bad export cookie 10303752529429420378 [12304.352765] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12304.365214] LustreError: 393097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12314.842802] Lustre: server umount lustre-MDT0001 complete [12322.913549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12327.761954] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12330.899732] LustreError: 394899:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12337.342757] Lustre: 395238:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.143@tcp [12339.677255] Lustre: server umount MGS complete [12353.091319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12353.666370] LustreError: 395962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.686252] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12353.692831] Lustre: Skipped 1 previous similar message [12353.735314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12359.141757] LustreError: 395963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12359.406213] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12369.377760] LustreError: 395962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12369.409215] LustreError: 395962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12369.627454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12369.971308] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12370.061460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12375.424442] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12384.843177] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12391.693484] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12394.135500] Lustre: Permanently reactivating lustre-OST0000 [12394.159374] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12394.163682] Lustre: Skipped 2 previous similar messages [12421.092761] LustreError: 397694:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12431.198841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12431.623317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12441.594401] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12449.616233] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12456.438880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12456.442595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12456.464925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12462.675696] Lustre: server umount lustre-OST0000 complete [12471.780367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12471.796033] Lustre: Skipped 2 previous similar messages [12473.857346] Lustre: server umount lustre-MDT0000 complete [12476.896827] LustreError: 399329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12476.915797] LustreError: 399329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12477.862053] LustreError: 395943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385332 with bad export cookie 10303752529429421435 [12477.873812] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12477.876096] LustreError: 395943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12478.229828] Lustre: server umount lustre-MDT0001 complete [12489.314941] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [12492.148814] Key type lgssc unregistered [12492.420761] LNet: 400284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12492.427055] LNetError: 400284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12493.478476] LNet: Removed LNI 192.168.202.143@tcp [12494.302798] Key type .llcrypt unregistered [12494.305981] Key type ._llcrypt unregistered [12511.909364] Key type ._llcrypt registered [12511.912318] Key type .llcrypt registered [12511.996256] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [12527.368161] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [12528.633858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12528.690143] alg: No test for adler32 (adler32-zlib) [12529.734431] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [12530.027525] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [12531.727224] Key type lgssc registered [12533.120762] Lustre: Echo OBD driver; http://www.lustre.org/ [12569.644369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12569.683689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12570.940162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12570.980727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12571.083534] Lustre: lustre-MDT0000: new disk, initializing [12571.219800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12571.240396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12575.907764] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12586.265247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12586.397247] Lustre: 403444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12586.470914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12586.475061] Lustre: Skipped 1 previous similar message [12586.574689] Lustre: lustre-MDT0001: new disk, initializing [12586.696350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12586.751136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12586.771270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12591.674947] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12602.542341] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12609.065602] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12616.743789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12617.001691] Lustre: lustre-OST0000: new disk, initializing [12617.012990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12617.036912] Lustre: 405414:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12617.165914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12623.478752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12623.496572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12623.666915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12624.331378] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12635.244955] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12641.079481] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12641.246556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12645.895318] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12646.115095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12649.459028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12649.466565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12649.470064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12649.476226] Lustre: Skipped 1 previous similar message [12649.485174] Lustre: Skipped 1 previous similar message [12654.565279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12654.576546] Lustre: Skipped 1 previous similar message [12659.695603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12662.239328] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12662.446923] Lustre: server umount lustre-OST0000 complete [12668.896363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12668.904343] LustreError: Skipped 1 previous similar message [12668.911181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12668.937457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12668.941945] Lustre: Skipped 1 previous similar message [12669.927097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12672.479436] Lustre: server umount lustre-MDT0000 complete [12675.712660] LustreError: 403437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385530 with bad export cookie 2543337304517194744 [12675.723623] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12675.729857] LustreError: 403437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12676.084574] Lustre: server umount lustre-MDT0001 complete [12685.931576] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12687.873478] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:59:01 (1787385541) [12694.912631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12695.409938] LustreError: 407875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12695.430643] LustreError: 407875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12695.493042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12699.595320] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12700.641156] LustreError: 407876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12705.779115] LustreError: 407875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.584903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12710.342471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12717.202409] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12726.481628] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12733.000773] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12740.849782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12741.097801] LustreError: 409697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12741.322504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12748.835369] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12757.008581] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12759.227157] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12761.724699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12770.283029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12770.304020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12770.306317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12770.321938] Lustre: Skipped 3 previous similar messages [12773.897224] Lustre: server umount lustre-OST0000 complete [12780.514280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12780.523382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12780.526357] Lustre: Skipped 2 previous similar messages [12792.287222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12793.140668] Lustre: server umount lustre-MDT0000 complete [12795.874464] LustreError: 408603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12795.899797] LustreError: 408603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12797.424690] LustreError: 407855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385652 with bad export cookie 2543337304517195626 [12797.441657] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12797.447859] LustreError: 407855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12797.790977] Lustre: server umount lustre-MDT0001 complete [12809.508305] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [12812.536528] Key type lgssc unregistered [12812.800395] LNet: 411617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12812.807749] LNetError: 411617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12813.863074] LNet: Removed LNI 192.168.202.143@tcp [12814.824614] Key type .llcrypt unregistered [12814.829303] Key type ._llcrypt unregistered [12840.938380] Key type ._llcrypt registered [12840.941349] Key type .llcrypt registered [12841.068502] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [12841.497432] Key type .llcrypt unregistered [12841.498569] Key type ._llcrypt unregistered [12860.565806] Key type ._llcrypt registered [12860.568772] Key type .llcrypt registered [12860.755256] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [12862.152389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12862.224387] alg: No test for adler32 (adler32-zlib) [12863.412873] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [12863.794199] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [12865.455370] Key type lgssc registered [12866.748560] Lustre: Echo OBD driver; http://www.lustre.org/ [12876.683798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12876.710804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12878.384788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12884.315871] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12888.549722] LustreError: 413836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12888.581650] LustreError: 413836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12893.689666] LustreError: 413837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12895.640397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12895.974169] LustreError: 413836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12896.026289] LustreError: 413836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12896.049893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12901.356770] LustreError: 413832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12901.394216] LustreError: 413832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12903.428790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12904.930976] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12908.517899] LustreError: 414580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12908.551906] LustreError: 414580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12916.434918] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12918.764762] LustreError: 413837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12918.781285] LustreError: 413837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12926.081620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12926.644150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12932.636050] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [12944.353734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12944.369317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12944.387060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12946.915481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12946.930499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12946.948205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12952.559663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12952.566512] Lustre: Skipped 2 previous similar messages [12957.667188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12957.682120] Lustre: Skipped 1 previous similar message [12958.175322] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12958.629659] Lustre: server umount lustre-OST0000 complete [12967.411295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12967.433710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12967.455530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12967.462865] Lustre: Skipped 1 previous similar message [12978.144466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12978.161061] Lustre: Skipped 3 previous similar messages [12979.167222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12979.698081] Lustre: server umount lustre-MDT0000 complete [12983.268429] LustreError: 413837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12983.279696] LustreError: 413837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12985.341233] LustreError: 413818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385840 with bad export cookie 12030095225600359811 [12985.359034] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12985.762664] Lustre: server umount lustre-MDT0001 complete [13000.375235] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [13004.063806] Key type lgssc unregistered [13004.470643] LNet: 416991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13004.487314] LNetError: 416991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13004.505080] LNet: Removed LNI 192.168.202.143@tcp [13005.662822] Key type .llcrypt unregistered [13005.667643] Key type ._llcrypt unregistered [13032.756384] Key type ._llcrypt registered [13032.759657] Key type .llcrypt registered [13032.916336] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [13034.212528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13034.332161] alg: No test for adler32 (adler32-zlib) [13035.529931] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [13035.859080] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [13037.515219] Key type lgssc registered [13038.968807] Lustre: Echo OBD driver; http://www.lustre.org/ [13048.003326] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13048.026782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13063.647547] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13068.768365] LustreError: 418387:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.143@tcp: failed processing log, type 4: rc = -110 [13095.391213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13104.204497] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13113.981268] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13120.727519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13121.379863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13121.490351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13126.625371] LustreError: 418413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13126.651718] LustreError: 418413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13128.686699] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13129.702328] LustreError: 418412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13129.739647] LustreError: 418412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13134.826124] LustreError: 419424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13134.860486] LustreError: 419424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13139.946162] LustreError: 419425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13139.975552] LustreError: 419425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13140.499554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13141.213361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13145.061638] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13154.361719] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13161.049343] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13171.683473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13171.686950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13171.705491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13172.198590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13172.223125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13172.230188] Lustre: Skipped 1 previous similar message [13175.760303] Lustre: server umount lustre-OST0000 complete [13181.924162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13181.937887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13181.958878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13182.776125] Lustre: server umount lustre-MDT0000 complete [13187.243275] LustreError: 418393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386041 with bad export cookie 9723746846765568229 [13187.263855] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13187.665311] Lustre: server umount lustre-MDT0001 complete [13200.731686] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [13203.931843] Key type lgssc unregistered [13204.279150] LNet: 421963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13204.286125] LNetError: 421963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13204.309297] LNet: Removed LNI 192.168.202.143@tcp [13205.484598] Key type .llcrypt unregistered [13205.486831] Key type ._llcrypt unregistered [13236.013344] Key type ._llcrypt registered [13236.019549] Key type .llcrypt registered [13236.156472] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [13237.932457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13238.015585] alg: No test for adler32 (adler32-zlib) [13239.155364] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [13239.474352] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [13241.265845] Key type lgssc registered [13242.964305] Lustre: Echo OBD driver; http://www.lustre.org/ [13252.436805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13252.480903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13253.998308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13259.511366] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13263.334398] LustreError: 423378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13263.354122] LustreError: 423378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13268.451325] LustreError: 423379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13270.280600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13270.851915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13276.751922] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13287.555389] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13294.815704] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13302.949128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13303.521380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13311.265573] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13322.097298] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13324.020554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13332.456545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13332.463947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13332.475406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13333.302323] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.43@tcp (stopping) [13334.504833] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13334.506654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13334.544306] Lustre: Skipped 1 previous similar message [13334.550615] Lustre: Skipped 1 previous similar message [13337.825284] Lustre: server umount lustre-MDT0000 complete [13338.352813] LustreError: 423374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13342.383757] LustreError: 423360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.43@tcp arrived at 1787386197 with bad export cookie 15032648184280467685 [13342.423877] LustreError: 423360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13344.737662] LustreError: 424108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13344.756286] LustreError: 424108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13345.308692] LustreError: 425225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386200 with bad export cookie 15032648184280467020 [13345.324420] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13345.349054] LustreError: 425225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13346.787644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13346.793458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13346.802946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13346.812182] Lustre: Skipped 1 previous similar message [13349.857361] LustreError: 423378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13349.875811] LustreError: 423378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13350.892782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13350.901184] Lustre: Skipped 1 previous similar message [13351.596588] Lustre: server umount lustre-OST0000 complete [13361.124035] LustreError: 423375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13361.148233] LustreError: 423375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13362.114806] Lustre: server umount lustre-MDT0001 complete [13375.190922] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [13378.816708] Key type lgssc unregistered [13379.257622] LNet: 427073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13379.268620] LNetError: 427073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13380.326367] LNet: Removed LNI 192.168.202.143@tcp [13381.492905] Key type .llcrypt unregistered [13381.497329] Key type ._llcrypt unregistered [13411.861502] Key type ._llcrypt registered [13411.863583] Key type .llcrypt registered [13411.961653] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [13413.851653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13413.916108] alg: No test for adler32 (adler32-zlib) [13415.227359] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [13415.574126] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [13417.287182] Key type lgssc registered [13418.885809] Lustre: Echo OBD driver; http://www.lustre.org/ [13430.983750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13431.004765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13432.562340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13439.347559] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13443.055478] LustreError: 428493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13443.093900] LustreError: 428493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13448.184099] LustreError: 428494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13450.571617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13451.249266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13457.004356] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13466.736319] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13473.761863] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13482.056108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13482.599556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13489.237582] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13497.220346] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13503.096615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13507.558088] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13507.569458] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13507.582129] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13508.373187] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.43@tcp (stopping) [13508.599683] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13513.451676] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.43@tcp (stopping) [13513.487468] Lustre: Skipped 2 previous similar messages [13518.615199] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.43@tcp (stopping) [13518.631687] Lustre: Skipped 2 previous similar messages [13520.352656] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13520.734742] Lustre: server umount lustre-MDT0001 complete [13523.940900] LustreError: 429222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13523.975820] LustreError: 429222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13529.058353] LustreError: 428493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13529.069226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13529.069862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13529.154297] Lustre: server umount lustre-OST0000 complete [13533.949512] Lustre: server umount lustre-MDT0000 complete [13550.894288] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [13554.219789] Key type lgssc unregistered [13554.541763] LNet: 432184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13554.549682] LNetError: 432184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13554.562754] LNet: Removed LNI 192.168.202.143@tcp [13555.595402] Key type .llcrypt unregistered [13555.597356] Key type ._llcrypt unregistered [13585.332264] Key type ._llcrypt registered [13585.335335] Key type .llcrypt registered [13585.452225] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [13601.969815] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [13603.120532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13603.234186] alg: No test for adler32 (adler32-zlib) [13604.533001] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [13604.761825] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [13606.439286] Key type lgssc registered [13607.859791] Lustre: Echo OBD driver; http://www.lustre.org/ [13651.541522] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [13668.039547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13668.063580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13669.317118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13669.343791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13669.401989] Lustre: lustre-MDT0000: new disk, initializing [13669.482415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13669.493483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13673.960235] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13690.594576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13690.922841] Lustre: lustre-OST0000: new disk, initializing [13690.926968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13690.931131] Lustre: Skipped 1 previous similar message [13690.938131] Lustre: 437221:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13691.015609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13697.707776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13697.729328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13697.775200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13697.873933] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13710.599277] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13726.090772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13726.244521] Lustre: 436202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13726.351305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13726.453765] Lustre: lustre-MDT0001: new disk, initializing [13726.586368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13726.627221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13726.643457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13731.263263] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13740.167638] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13740.427936] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13746.321509] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13746.596362] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13753.827954] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13753.831160] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13753.854209] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13753.868136] Lustre: Skipped 1 previous similar message [13755.656285] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.43@tcp (stopping) [13758.674186] Lustre: server umount lustre-MDT0001 complete [13764.077587] LustreError: 436213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13764.091419] LustreError: 436213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13769.192349] LustreError: 436214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13769.200809] LustreError: 436214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13774.309429] LustreError: 436209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13774.335354] LustreError: 436209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13779.425332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13779.427343] LustreError: 436214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13779.452775] Lustre: Skipped 1 previous similar message [13779.463603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13782.844953] Lustre: server umount lustre-OST0000 complete [13784.552207] LustreError: 438561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13793.917365] Lustre: server umount lustre-MDT0000 complete [13810.000540] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [13813.319157] Key type lgssc unregistered [13813.633722] LNet: 440974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13813.643349] LNetError: 440974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13813.654749] LNet: Removed LNI 192.168.202.143@tcp [13814.587966] Key type .llcrypt unregistered [13814.593956] Key type ._llcrypt unregistered [13856.413769] Key type ._llcrypt registered [13856.416693] Key type .llcrypt registered [13856.574498] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [13856.933447] Key type .llcrypt unregistered [13856.936941] Key type ._llcrypt unregistered [13873.347438] Key type ._llcrypt registered [13873.349470] Key type .llcrypt registered [13873.462988] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [13874.870948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13874.993650] alg: No test for adler32 (adler32-zlib) [13876.158267] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [13876.379691] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [13878.039164] Key type lgssc registered [13879.262134] Lustre: Echo OBD driver; http://www.lustre.org/ [13909.363357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13909.431453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13910.734715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13910.761443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13910.834322] Lustre: lustre-MDT0000: new disk, initializing [13910.903372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13910.922132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13916.008605] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13927.799154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13927.893329] Lustre: 444355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13927.939131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13927.942667] Lustre: Skipped 1 previous similar message [13928.011838] Lustre: lustre-MDT0001: new disk, initializing [13928.218369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13928.256587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13928.265746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13933.296814] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13944.243175] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13951.677672] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13960.033336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13960.352965] Lustre: lustre-OST0000: new disk, initializing [13960.357286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13960.375305] Lustre: 446324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13960.489098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13968.322854] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [13969.013265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13969.026378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13969.219432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13979.186834] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13989.858216] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13989.867644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13989.877281] Lustre: Skipped 1 previous similar message [13989.877750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13989.906852] Lustre: Skipped 2 previous similar messages [13992.617901] Lustre: server umount lustre-MDT0000 complete [13996.664764] LustreError: 444348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386851 with bad export cookie 638778892874866484 [13996.667375] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13996.673028] LustreError: 444348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13997.051410] Lustre: server umount lustre-MDT0001 complete [14011.515637] Lustre: server umount lustre-OST0000 complete [14023.677547] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:21:16 (1787386876) [14033.723150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14034.521732] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14034.559411] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14034.721590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14040.047280] LustreError: 448355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14041.914432] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [14045.163543] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14051.298544] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14051.326625] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14054.320841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14054.896208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14059.646717] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [14069.075846] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14074.923739] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14095.210241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14095.487539] Lustre: Found index 0 for lustre-OST0000, updating log [14095.678661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14102.293593] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [14106.933324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14110.801941] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14117.345076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14117.360760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14117.379468] Lustre: Skipped 1 previous similar message [14117.394764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14118.434757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14118.446844] Lustre: Skipped 1 previous similar message [14122.844516] Lustre: server umount lustre-OST0000 complete [14139.873156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14139.877178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14139.891759] Lustre: Skipped 2 previous similar messages [14139.903591] Lustre: Skipped 1 previous similar message [14144.661709] Lustre: server umount lustre-MDT0000 complete [14144.999660] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14145.014773] LustreError: 448354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14148.346573] LustreError: 448336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387003 with bad export cookie 638778892874867870 [14148.360593] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14148.663331] Lustre: server umount lustre-MDT0001 complete [14153.892617] LNet: 1 local NIs in recovery (showing 1): 192.168.202.143@tcp [14158.962286] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:23:32 (1787387012) [14784.183099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14784.566591] LustreError: 454767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14784.582228] LustreError: 454767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14784.633878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14788.421705] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [14790.121315] LustreError: 454768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14795.236063] LustreError: 454767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14795.766468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14796.304129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14800.431790] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [14808.782934] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14817.937161] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14841.208119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14841.728455] Lustre: Found index 0 for lustre-OST0000, updating log [14842.069981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14844.064386] LNet: 1 local NIs in recovery (showing 1): 192.168.202.143@tcp [14849.680078] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [14858.789435] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14862.317778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14862.345976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14862.350021] Lustre: Skipped 1 previous similar message [14867.696615] Lustre: server umount lustre-OST0000 complete [14873.061233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14873.068375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14873.094443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14874.151994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14874.160693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14876.258986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14876.272538] Lustre: Skipped 3 previous similar messages [14878.910824] Lustre: server umount lustre-MDT0000 complete [14879.337196] LustreError: 454768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14879.369229] LustreError: 454768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14883.644166] LustreError: 454749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387738 with bad export cookie 638778892874868955 [14883.648686] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14883.670862] LustreError: 454749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [14884.271939] Lustre: server umount lustre-MDT0001 complete [14903.456317] LNet: 1 local NIs in recovery (showing 1): 192.168.202.143@tcp [14916.182334] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [14927.566769] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [14967.484565] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:37:01 (1787387821) [14986.779695] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [14989.412258] Key type lgssc unregistered [14989.798781] LNet: 461394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14989.808238] LNetError: 461394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15103.534982] LNet: Removed LNI 192.168.202.143@tcp [15104.311097] Key type .llcrypt unregistered [15104.312995] Key type ._llcrypt unregistered [15122.256405] Key type ._llcrypt registered [15122.269034] Key type .llcrypt registered [15122.423840] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [15124.358860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15124.690980] alg: No test for adler32 (adler32-zlib) [15125.798747] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [15126.108578] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [15127.783193] Key type lgssc registered [15128.992718] Lustre: Echo OBD driver; http://www.lustre.org/ [15133.937681] LNet: Added LNI 192.168.202.143@tcp100 [8/256/0/180] [15134.868099] LNet: Added LNI 192.168.202.143@tcp101 [8/256/0/180] [15136.041474] LNet: Added LNI 192.168.202.143@tcp102 [8/256/0/180] [15137.226605] LNet: Added LNI 192.168.202.143@tcp103 [8/256/0/180] [15138.598443] LNet: Added LNI 192.168.202.143@tcp104 [8/256/0/180] [15139.939373] LNet: Added LNI 192.168.202.143@tcp105 [8/256/0/180] [15141.316594] LNet: Added LNI 192.168.202.143@tcp106 [8/256/0/180] [15142.521124] LNet: Added LNI 192.168.202.143@tcp107 [8/256/0/180] [15143.537739] LNet: Added LNI 192.168.202.143@tcp108 [8/256/0/180] [15144.529816] LNet: Added LNI 192.168.202.143@tcp109 [8/256/0/180] [15145.679715] LNet: Added LNI 192.168.202.143@tcp110 [8/256/0/180] [15146.605282] LNet: Added LNI 192.168.202.143@tcp111 [8/256/0/180] [15147.518155] LNet: Added LNI 192.168.202.143@tcp112 [8/256/0/180] [15148.394950] LNet: Added LNI 192.168.202.143@tcp113 [8/256/0/180] [15149.441847] LNet: Added LNI 192.168.202.143@tcp114 [8/256/0/180] [15150.506272] LNet: Added LNI 192.168.202.143@tcp115 [8/256/0/180] [15151.288928] LNet: Added LNI 192.168.202.143@tcp116 [8/256/0/180] [15152.248463] LNet: Added LNI 192.168.202.143@tcp117 [8/256/0/180] [15153.280768] LNet: Added LNI 192.168.202.143@tcp118 [8/256/0/180] [15154.273945] LNet: Added LNI 192.168.202.143@tcp119 [8/256/0/180] [15155.129051] LNet: Added LNI 192.168.202.143@tcp120 [8/256/0/180] [15156.058525] LNet: Added LNI 192.168.202.143@tcp121 [8/256/0/180] [15156.950244] LNet: Added LNI 192.168.202.143@tcp122 [8/256/0/180] [15158.028607] LNet: Added LNI 192.168.202.143@tcp123 [8/256/0/180] [15159.178865] LNet: Added LNI 192.168.202.143@tcp124 [8/256/0/180] [15160.243167] LNet: Added LNI 192.168.202.143@tcp125 [8/256/0/180] [15161.413829] LNet: Added LNI 192.168.202.143@tcp126 [8/256/0/180] [15162.223147] LNet: Added LNI 192.168.202.143@tcp127 [8/256/0/180] [15163.230036] LNet: Added LNI 192.168.202.143@tcp128 [8/256/0/180] [15164.232577] LNet: Added LNI 192.168.202.143@tcp129 [8/256/0/180] [15165.288939] LNet: Added LNI 192.168.202.143@tcp130 [8/256/0/180] [15166.470225] LNet: Added LNI 192.168.202.143@tcp131 [8/256/0/180] [15167.650410] LNet: Added LNI 192.168.202.143@tcp132 [8/256/0/180] [15168.669216] LNet: Added LNI 192.168.202.143@tcp133 [8/256/0/180] [15169.628772] LNet: Added LNI 192.168.202.143@tcp134 [8/256/0/180] [15170.580460] LNet: Added LNI 192.168.202.143@tcp135 [8/256/0/180] [15178.866291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15178.882647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15180.078555] Lustre: 464428:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.143@tcp: can't add NID 192.168.202.143@tcp130: rc = -75 [15180.231031] LNetError: 461886:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.143 [15180.247915] LNetError: Protocol error connecting to 192.168.202.143@tcp114 at host 192.168.202.143:988: is it running a compatible version of Lustre? [15180.264664] LNetError: Skipped 2 previous similar messages [15180.357529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15180.385959] LNet: 11 local NIs in recovery (showing 11): 192.168.202.143@tcp119, 192.168.202.143@tcp120, 192.168.202.143@tcp121, 192.168.202.143@tcp123, 192.168.202.143@tcp122, 192.168.202.143@tcp124, 192.168.202.143@tcp125, 192.168.202.143@tcp126, 192.168.202.143@tcp127, 192.168.202.143@tcp128, 192.168.202.143@tcp129 [15180.437455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15180.604205] Lustre: lustre-MDT0000: new disk, initializing [15180.807463] Lustre: 464485:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.143@tcp: can't add NID 192.168.202.143@tcp130: rc = -75 [15180.836286] Lustre: 464485:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15180.869599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15180.911467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15181.472702] LNetError: 464461:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.143 [15181.495951] LNetError: 464461:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [15181.520691] LNetError: Protocol error connecting to 192.168.202.143@tcp119 at host 192.168.202.143:988: is it running a compatible version of Lustre? [15183.521814] LNetError: 461889:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.143 [15183.550212] LNetError: 461889:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15183.569039] LNetError: Protocol error connecting to 192.168.202.143@tcp119 at host 192.168.202.143:988: is it running a compatible version of Lustre? [15183.585215] LNetError: Skipped 3 previous similar messages [15187.620177] LNetError: 464452:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.143 [15187.643396] LNetError: Protocol error connecting to 192.168.202.143@tcp119 at host 192.168.202.143:988: is it running a compatible version of Lustre? [15187.728409] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [15195.808459] LNetError: 461888:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.143 [15195.825730] LNetError: Protocol error connecting to 192.168.202.143@tcp119 at host 192.168.202.143:988: is it running a compatible version of Lustre? [15200.972277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15201.166371] Lustre: 464446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15201.234748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15201.238514] Lustre: Skipped 1 previous similar message [15201.347743] Lustre: lustre-MDT0001: new disk, initializing [15201.381335] Lustre: 465249:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.143@tcp: can't add NID 192.168.202.143@tcp130: rc = -75 [15201.389247] Lustre: 465249:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15201.467323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15201.491344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15201.508456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15207.654796] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [15208.527530] Lustre: 465538:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.143@tcp: can't add NID 192.168.202.143@tcp130: rc = -75 [15208.546025] Lustre: 465538:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15210.080767] LNetError: 464460:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.143 [15210.089393] LNetError: Protocol error connecting to 192.168.202.143@tcp116 at host 192.168.202.143:988: is it running a compatible version of Lustre? [15218.873491] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15225.889449] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15226.466562] LNetError: 464452:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.143 [15226.482642] LNetError: 464452:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15226.493293] LNetError: Protocol error connecting to 192.168.202.143@tcp114 at host 192.168.202.143:988: is it running a compatible version of Lustre? [15226.514637] LNetError: Skipped 9 previous similar messages [15233.547763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15233.781871] Lustre: lustre-OST0000: new disk, initializing [15233.784783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15233.790519] Lustre: 466423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15233.898378] Lustre: 466428:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.143@tcp: can't add NID 192.168.202.143@tcp130: rc = -75 [15233.905504] Lustre: 466428:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15233.947408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15240.875096] LNet: 1 local NIs in recovery (showing 1): 192.168.202.143@tcp119 [15241.788438] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [15242.351602] Lustre: 466794:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.143@tcp: can't add NID 192.168.202.143@tcp130: rc = -75 [15242.369879] Lustre: 466794:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15242.454069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15242.477117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15242.732075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15255.129326] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15259.234246] LNetError: 464461:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.143 [15259.239522] LNetError: 464461:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [15259.250052] LNetError: Protocol error connecting to 192.168.202.143@tcp118 at host 192.168.202.143:988: is it running a compatible version of Lustre? [15259.262147] LNetError: Skipped 18 previous similar messages [15268.334290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15268.341143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15268.355349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15268.370189] Lustre: Skipped 1 previous similar message [15268.395164] Lustre: Skipped 1 previous similar message [15271.637881] Lustre: server umount lustre-OST0000 complete [15278.560752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15278.561882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15278.599069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15278.605726] LustreError: Skipped 1 previous similar message [15278.632556] Lustre: Skipped 1 previous similar message [15283.546114] Lustre: server umount lustre-MDT0000 complete [15288.189117] LustreError: 464437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388142 with bad export cookie 14130263477062516875 [15288.191464] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15288.197588] LustreError: 464437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15288.699459] Lustre: server umount lustre-MDT0001 complete [15301.832967] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [15305.650622] Key type lgssc unregistered [15306.023818] LNet: 468341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15306.040454] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15306.048531] LNet: Removed LNI 192.168.202.143@tcp [15307.103428] LNet: Removed LNI 192.168.202.143@tcp100 [15307.116601] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15307.138401] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15307.147861] LNet: Removed LNI 192.168.202.143@tcp101 [15308.191241] LNet: Removed LNI 192.168.202.143@tcp102 [15308.193388] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15308.207894] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15308.219794] LNet: Removed LNI 192.168.202.143@tcp103 [15309.279338] LNet: Removed LNI 192.168.202.143@tcp104 [15310.313040] LNet: Removed LNI 192.168.202.143@tcp105 [15310.353611] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15310.366900] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15310.377740] LNet: Removed LNI 192.168.202.143@tcp106 [15311.391458] LNet: Removed LNI 192.168.202.143@tcp107 [15312.415340] LNet: Removed LNI 192.168.202.143@tcp108 [15313.439875] LNet: Removed LNI 192.168.202.143@tcp109 [15314.463867] LNet: Removed LNI 192.168.202.143@tcp110 [15314.471846] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15314.478632] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15315.551155] LNet: Removed LNI 192.168.202.143@tcp111 [15316.575153] LNet: Removed LNI 192.168.202.143@tcp112 [15317.601266] LNet: Removed LNI 192.168.202.143@tcp113 [15318.623176] LNet: Removed LNI 192.168.202.143@tcp114 [15319.648338] LNet: Removed LNI 192.168.202.143@tcp115 [15320.672176] LNet: Removed LNI 192.168.202.143@tcp116 [15321.695173] LNet: Removed LNI 192.168.202.143@tcp117 [15322.719349] LNet: Removed LNI 192.168.202.143@tcp118 [15322.727277] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15322.754278] LNetError: 468341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [15322.766116] LNet: Removed LNI 192.168.202.143@tcp119 [15323.807486] LNet: Removed LNI 192.168.202.143@tcp120 [15324.895204] LNet: Removed LNI 192.168.202.143@tcp121 [15325.919183] LNet: Removed LNI 192.168.202.143@tcp122 [15326.943334] LNet: Removed LNI 192.168.202.143@tcp123 [15327.967260] LNet: Removed LNI 192.168.202.143@tcp124 [15329.055194] LNet: Removed LNI 192.168.202.143@tcp125 [15330.079202] LNet: Removed LNI 192.168.202.143@tcp126 [15331.103197] LNet: Removed LNI 192.168.202.143@tcp127 [15332.127705] LNet: Removed LNI 192.168.202.143@tcp128 [15333.151600] LNet: Removed LNI 192.168.202.143@tcp129 [15333.153470] LNet: Removed LNI 192.168.202.143@tcp130 [15333.170169] LNet: Removed LNI 192.168.202.143@tcp131 [15333.178577] LNet: Removed LNI 192.168.202.143@tcp132 [15333.192877] LNet: Removed LNI 192.168.202.143@tcp133 [15333.204820] LNet: Removed LNI 192.168.202.143@tcp134 [15333.248424] LNet: Removed LNI 192.168.202.143@tcp135 [15334.373678] Key type .llcrypt unregistered [15334.375411] Key type ._llcrypt unregistered [15361.965217] Key type ._llcrypt registered [15361.967133] Key type .llcrypt registered [15362.060276] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [15377.669559] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [15378.837362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15378.894398] alg: No test for adler32 (adler32-zlib) [15380.027994] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [15380.266951] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [15381.935851] Key type lgssc registered [15383.311880] Lustre: Echo OBD driver; http://www.lustre.org/ [15426.280741] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:44:39 (1787388279) [15448.188125] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [15451.119619] Key type lgssc unregistered [15451.348821] LNet: 472438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15451.353537] LNetError: 472438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15451.372872] LNet: Removed LNI 192.168.202.143@tcp [15452.095521] Key type .llcrypt unregistered [15452.097116] Key type ._llcrypt unregistered [15473.209308] Key type ._llcrypt registered [15473.215496] Key type .llcrypt registered [15473.407981] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [15475.029987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15475.094556] alg: No test for adler32 (adler32-zlib) [15476.326827] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [15476.657989] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [15478.455195] Key type lgssc registered [15479.976969] Lustre: Echo OBD driver; http://www.lustre.org/ [15492.509741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15492.591864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15493.967791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15494.027637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15494.186396] Lustre: lustre-MDT0000: new disk, initializing [15494.335418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15494.463954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15500.368412] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [15514.349694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15514.490891] Lustre: 473651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15514.548647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15514.553755] Lustre: Skipped 1 previous similar message [15514.681263] Lustre: lustre-MDT0001: new disk, initializing [15514.836157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15514.898285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15514.915872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15520.666530] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [15533.085952] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15540.129403] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15548.290972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15548.685977] Lustre: lustre-OST0000: new disk, initializing [15548.689337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15548.694773] Lustre: 475623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15548.794978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15556.738172] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [15557.767525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15557.781527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15557.911886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15569.727124] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15574.559573] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing lnet_if_list [15581.018447] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15581.057859] LNet: Added LNI 192.168.202.143@tcp10 [8/256/0/180] [15589.889550] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15589.915878] LNet: Added LNI 192.168.202.143@tcp11 [8/256/0/180] [15595.758692] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15595.786776] LNet: Added LNI 192.168.202.143@tcp12 [8/256/0/180] [15601.278130] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15601.305992] LNet: Added LNI 192.168.202.143@tcp13 [8/256/0/180] [15607.033898] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15607.078828] LNet: Added LNI 192.168.202.143@tcp14 [8/256/0/180] [15612.591647] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15612.635533] LNet: Added LNI 192.168.202.143@tcp15 [8/256/0/180] [15618.230832] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15618.252947] LNet: Added LNI 192.168.202.143@tcp16 [8/256/0/180] [15624.694876] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15624.742625] LNet: Added LNI 192.168.202.143@tcp17 [8/256/0/180] [15630.839369] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15630.892427] LNet: Added LNI 192.168.202.143@tcp18 [8/256/0/180] [15637.729301] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15637.773582] LNet: Added LNI 192.168.202.143@tcp19 [8/256/0/180] [15644.612745] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15644.640551] LNet: Added LNI 192.168.202.143@tcp20 [8/256/0/180] [15650.938433] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15650.971428] LNet: Added LNI 192.168.202.143@tcp21 [8/256/0/180] [15657.611094] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15657.680331] LNet: Added LNI 192.168.202.143@tcp22 [8/256/0/180] [15665.120140] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15665.168608] LNet: Added LNI 192.168.202.143@tcp23 [8/256/0/180] [15672.307678] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15672.336541] LNet: Added LNI 192.168.202.143@tcp24 [8/256/0/180] [15680.209346] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15680.261200] LNet: Added LNI 192.168.202.143@tcp25 [8/256/0/180] [15687.790299] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15687.843718] LNet: Added LNI 192.168.202.143@tcp26 [8/256/0/180] [15694.517530] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15694.538868] LNet: Added LNI 192.168.202.143@tcp27 [8/256/0/180] [15700.987309] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15701.008674] LNet: Added LNI 192.168.202.143@tcp28 [8/256/0/180] [15709.366837] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15709.411505] LNet: Added LNI 192.168.202.143@tcp29 [8/256/0/180] [15717.906338] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15717.943385] LNet: Added LNI 192.168.202.143@tcp30 [8/256/0/180] [15726.596634] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15726.618428] LNet: Added LNI 192.168.202.143@tcp31 [8/256/0/180] [15734.983857] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15735.017310] LNet: Added LNI 192.168.202.143@tcp32 [8/256/0/180] [15744.016179] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15744.075377] LNet: Added LNI 192.168.202.143@tcp33 [8/256/0/180] [15752.815297] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15752.857914] LNet: Added LNI 192.168.202.143@tcp34 [8/256/0/180] [15761.747397] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15761.776615] LNet: Added LNI 192.168.202.143@tcp35 [8/256/0/180] [15768.967148] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15769.011324] LNet: Added LNI 192.168.202.143@tcp36 [8/256/0/180] [15777.045351] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15777.118715] LNet: Added LNI 192.168.202.143@tcp37 [8/256/0/180] [15787.192733] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15787.230325] LNet: Added LNI 192.168.202.143@tcp38 [8/256/0/180] [15797.613388] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15797.680250] LNet: Added LNI 192.168.202.143@tcp39 [8/256/0/180] [15806.644409] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15806.730927] LNet: Added LNI 192.168.202.143@tcp40 [8/256/0/180] [15815.182401] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15815.195280] LNet: Added LNI 192.168.202.143@tcp41 [8/256/0/180] [15822.797777] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15822.816963] LNet: Added LNI 192.168.202.143@tcp42 [8/256/0/180] [15832.073924] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15832.125783] LNet: Added LNI 192.168.202.143@tcp43 [8/256/0/180] [15841.138544] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15841.184978] LNet: Added LNI 192.168.202.143@tcp44 [8/256/0/180] [15851.466220] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15851.494685] LNet: Added LNI 192.168.202.143@tcp45 [8/256/0/180] [15863.306590] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15863.337264] LNet: Added LNI 192.168.202.143@tcp46 [8/256/0/180] [15872.160664] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15872.218591] LNet: Added LNI 192.168.202.143@tcp47 [8/256/0/180] [15884.016325] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15884.089961] LNet: Added LNI 192.168.202.143@tcp48 [8/256/0/180] [15894.246569] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15894.305647] LNet: Added LNI 192.168.202.143@tcp49 [8/256/0/180] [15908.903479] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15908.936555] LNet: Added LNI 192.168.202.143@tcp50 [8/256/0/180] [15934.947839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15934.952796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15934.960087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15936.994324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15937.000920] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15937.020544] Lustre: Skipped 1 previous similar message [15937.039277] Lustre: Skipped 1 previous similar message [15942.115150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15942.124287] Lustre: Skipped 3 previous similar messages [15947.234805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15947.248300] Lustre: Skipped 2 previous similar messages [15949.280657] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15949.492488] Lustre: server umount lustre-MDT0000 complete [15953.554960] LustreError: 475622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388808 with bad export cookie 2528386813730545621 [15953.558353] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15953.567629] LustreError: 475622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15954.390645] Lustre: server umount lustre-MDT0001 complete [15973.856751] Lustre: 472927:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388812/real 1787388812] req@00000000584358d5 x1874212506768256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15973.919418] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15974.846478] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [15974.886589] Lustre: 472924:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388813/real 1787388813] req@00000000b8f558b1 x1874212506768384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15979.045450] Lustre: 472924:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388817/real 1787388817] req@0000000019f988b9 x1874212506768512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15985.251431] Lustre: 472927:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388823/real 1787388823] req@00000000d0debbb6 x1874212506768896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15985.271502] Lustre: 472927:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15985.996317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15990.239397] Lustre: 472925:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388828/real 1787388828] req@00000000ccab5ddb x1874212506769024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15998.559890] LustreError: 472923:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000040869de2 x1874212506769664/t0(0) o250->MGC192.168.202.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15998.948411] LustreError: 475615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15998.961194] LustreError: 475615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15999.009278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16004.064877] LustreError: 475616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16004.508443] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16008.171468] LustreError: 475615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16008.183289] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16013.291083] LustreError: 475616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16014.656866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16015.192852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16019.310800] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16020.385995] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16029.311929] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16035.397544] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16061.412471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16061.419620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16061.446803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16065.504449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16065.521068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16067.564907] Lustre: server umount lustre-OST0000 complete [16075.748263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16075.769184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16075.780090] Lustre: Skipped 3 previous similar messages [16078.648820] Lustre: server umount lustre-MDT0000 complete [16080.866965] LustreError: 499896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16080.884760] LustreError: 499896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16083.390355] LustreError: 475622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388938 with bad export cookie 2528386813730548386 [16083.405821] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16083.745603] Lustre: server umount lustre-MDT0001 complete [16097.446372] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [16100.858662] Key type lgssc unregistered [16101.301279] LNet: 502701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16101.314346] LNetError: 502701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16101.321604] LNet: Removed LNI 192.168.202.143@tcp [16101.325338] LNet: Removed LNI 192.168.202.143@tcp10 [16101.329326] LNet: Removed LNI 192.168.202.143@tcp11 [16101.332294] LNet: Removed LNI 192.168.202.143@tcp12 [16101.334719] LNet: Removed LNI 192.168.202.143@tcp13 [16101.338937] LNet: Removed LNI 192.168.202.143@tcp14 [16101.345817] LNet: Removed LNI 192.168.202.143@tcp15 [16101.348618] LNet: Removed LNI 192.168.202.143@tcp16 [16101.352985] LNet: Removed LNI 192.168.202.143@tcp17 [16101.357185] LNet: Removed LNI 192.168.202.143@tcp18 [16101.361258] LNet: Removed LNI 192.168.202.143@tcp19 [16101.365464] LNet: Removed LNI 192.168.202.143@tcp20 [16101.367474] LNet: Removed LNI 192.168.202.143@tcp21 [16101.372280] LNet: Removed LNI 192.168.202.143@tcp22 [16101.375478] LNet: Removed LNI 192.168.202.143@tcp23 [16101.379286] LNet: Removed LNI 192.168.202.143@tcp24 [16101.381629] LNet: Removed LNI 192.168.202.143@tcp25 [16101.384942] LNet: Removed LNI 192.168.202.143@tcp26 [16101.388178] LNet: Removed LNI 192.168.202.143@tcp27 [16101.390571] LNet: Removed LNI 192.168.202.143@tcp28 [16101.394312] LNet: Removed LNI 192.168.202.143@tcp29 [16101.398414] LNet: Removed LNI 192.168.202.143@tcp30 [16101.400496] LNet: Removed LNI 192.168.202.143@tcp31 [16101.403471] LNet: Removed LNI 192.168.202.143@tcp32 [16101.405545] LNet: Removed LNI 192.168.202.143@tcp33 [16101.407766] LNet: Removed LNI 192.168.202.143@tcp34 [16101.411203] LNet: Removed LNI 192.168.202.143@tcp35 [16101.414733] LNet: Removed LNI 192.168.202.143@tcp36 [16101.417518] LNet: Removed LNI 192.168.202.143@tcp37 [16101.421712] LNet: Removed LNI 192.168.202.143@tcp38 [16101.424750] LNet: Removed LNI 192.168.202.143@tcp39 [16101.427350] LNet: Removed LNI 192.168.202.143@tcp40 [16101.434628] LNet: Removed LNI 192.168.202.143@tcp41 [16101.438517] LNet: Removed LNI 192.168.202.143@tcp42 [16101.441413] LNet: Removed LNI 192.168.202.143@tcp43 [16101.443566] LNet: Removed LNI 192.168.202.143@tcp44 [16101.447472] LNet: Removed LNI 192.168.202.143@tcp45 [16101.450403] LNet: Removed LNI 192.168.202.143@tcp46 [16101.453829] LNet: Removed LNI 192.168.202.143@tcp47 [16101.458297] LNet: Removed LNI 192.168.202.143@tcp48 [16101.460738] LNet: Removed LNI 192.168.202.143@tcp49 [16101.470722] LNet: Removed LNI 192.168.202.143@tcp50 [16102.257273] Key type .llcrypt unregistered [16102.259974] Key type ._llcrypt unregistered [16125.508073] Key type ._llcrypt registered [16125.509713] Key type .llcrypt registered [16125.684817] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [16142.318958] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [16143.642347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16143.812889] alg: No test for adler32 (adler32-zlib) [16145.145777] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [16145.497311] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [16147.295176] Key type lgssc registered [16148.938923] Lustre: Echo OBD driver; http://www.lustre.org/ [16194.685843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16194.720748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16196.055205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16196.094413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16196.194066] Lustre: lustre-MDT0000: new disk, initializing [16196.313692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16196.323960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16202.918914] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16216.118171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16216.398936] Lustre: 505908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16216.569617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16216.584373] Lustre: Skipped 1 previous similar message [16216.765773] Lustre: lustre-MDT0001: new disk, initializing [16216.897619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16216.958327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16216.970174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16223.107554] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16234.481256] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16242.764179] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16253.343973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16253.723231] Lustre: *** cfs_fail_loc=90f, val=0*** [16253.890451] Lustre: lustre-OST0000: new disk, initializing [16253.895474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16253.916089] Lustre: 507924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16254.072508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16259.705625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16259.724098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16260.006964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16262.118873] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16275.207720] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16285.685747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16285.695898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16285.704825] Lustre: Skipped 1 previous similar message [16285.705719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16285.738502] LustreError: Skipped 1 previous similar message [16290.789234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16290.808139] Lustre: Skipped 2 previous similar messages [16291.407212] Lustre: server umount lustre-OST0000 complete [16295.904805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16295.908734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16295.921659] Lustre: Skipped 1 previous similar message [16301.024662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16301.031449] Lustre: Skipped 2 previous similar messages [16306.148098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16306.159372] Lustre: Skipped 1 previous similar message [16310.239306] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16310.421618] Lustre: server umount lustre-MDT0000 complete [16315.024305] LustreError: 507934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389169 with bad export cookie 1057885663578915707 [16315.026057] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16315.053552] LustreError: 507934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16315.616209] Lustre: server umount lustre-MDT0001 complete [16330.439781] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [16333.327466] Key type lgssc unregistered [16333.663644] LNet: 509837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16333.670140] LNetError: 509837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16333.697179] LNet: Removed LNI 192.168.202.143@tcp [16334.821310] Key type .llcrypt unregistered [16334.823226] Key type ._llcrypt unregistered [16361.314823] Key type ._llcrypt registered [16361.318716] Key type .llcrypt registered [16361.430742] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [16362.881029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16362.906623] alg: No test for adler32 (adler32-zlib) [16364.180478] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [16364.511778] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [16366.216175] Key type lgssc registered [16367.477709] Lustre: Echo OBD driver; http://www.lustre.org/ [16377.383169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16377.406710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16379.100191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16384.968887] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16388.581584] LustreError: 511108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16388.613484] LustreError: 511108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16393.704640] LustreError: 511109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16395.995215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16396.524613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16402.611824] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16411.863580] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16419.598084] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16427.824349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16428.114731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16436.125121] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16445.999948] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16462.287496] loop0: detected capacity change from 0 to 204800000 [16462.620245] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16462.642496] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16462.657473] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16472.348162] loop0: detected capacity change from 0 to 204800000 [16472.597389] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16472.616449] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16472.635660] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16480.137950] loop0: detected capacity change from 0 to 204800000 [16490.179313] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16490.408708] Lustre: fs15246-OST0000: new disk, initializing [16490.415246] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16490.421531] Lustre: 514718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16490.520700] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16498.430563] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16510.134318] loop1: detected capacity change from 0 to 204800000 [16517.500717] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16517.610704] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16517.682602] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16517.758184] Lustre: fs15246-MDT0000: new disk, initializing [16517.941406] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16517.982633] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16518.103693] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16519.140286] Lustre: 515841:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000808b51e7 x1874213437757696/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787389389 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16523.505273] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16528.243354] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16528.253066] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16528.285705] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16550.891463] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16550.913788] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16552.802862] Lustre: server umount fs15246-MDT0000 complete [16556.026278] LustreError: 511105:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16561.128698] LustreError: 512720:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16563.823173] Lustre: server umount fs15246-OST0000 complete [16576.992145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16576.995051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16577.005687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16580.065205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16580.079927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16580.100182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16586.721112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16586.738176] Lustre: Skipped 3 previous similar messages [16591.327663] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16591.701451] Lustre: server umount lustre-OST0000 complete [16596.452576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16596.459896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16596.479526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16602.842358] Lustre: server umount lustre-MDT0000 complete [16607.201725] LustreError: 512720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16607.219950] LustreError: 512720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16607.312369] LustreError: 511091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389462 with bad export cookie 14986764850572394335 [16607.319028] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16607.323441] LustreError: 511091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16607.912226] Lustre: server umount lustre-MDT0001 complete [16621.439711] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [16624.697650] Key type lgssc unregistered [16625.076747] LNet: 518943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16625.088131] LNetError: 518943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16625.104463] LNet: Removed LNI 192.168.202.143@tcp [16625.987312] Key type .llcrypt unregistered [16625.989402] Key type ._llcrypt unregistered [16652.254876] Key type ._llcrypt registered [16652.257269] Key type .llcrypt registered [16652.400530] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [16653.803513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16653.915228] alg: No test for adler32 (adler32-zlib) [16655.200759] Lustre: Lustre: Build Version: 2.17.57_47_gae0d133 [16655.536787] LNet: Added LNI 192.168.202.143@tcp [8/256/0/180] [16657.295270] Key type lgssc registered [16659.151374] Lustre: Echo OBD driver; http://www.lustre.org/ [16707.748506] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [16722.350542] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [16771.689878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16771.740188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16773.120210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16773.155846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16773.321283] Lustre: lustre-MDT0000: new disk, initializing [16773.439267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16773.485154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16779.590951] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16794.455909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16794.624787] Lustre: 523419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16794.706254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16794.710818] Lustre: Skipped 1 previous similar message [16794.798908] Lustre: lustre-MDT0001: new disk, initializing [16795.080437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16795.124098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16795.141561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16801.482676] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16813.962339] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16821.701914] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16831.539438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16831.872724] Lustre: lustre-OST0000: new disk, initializing [16831.876452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16831.887955] Lustre: 525390:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16832.048283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16838.229952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16838.237285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16838.417410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16840.368925] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_default_debug -1 all [16853.179781] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16859.453457] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16859.648211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16865.560551] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16865.866513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16869.344797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16869.367846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16869.372401] LustreError: Skipped 1 previous similar message [16869.389042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16869.392941] Lustre: Skipped 1 previous similar message [16869.426515] Lustre: Skipped 1 previous similar message [16874.470865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16874.879175] Lustre: server umount lustre-OST0000 complete [16882.151940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16882.163660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16882.181344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16882.200101] Lustre: Skipped 1 previous similar message [16884.707799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16884.709667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16886.230947] Lustre: server umount lustre-MDT0000 complete [16891.391267] LustreError: 523412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389746 with bad export cookie 16607037771649041319 [16891.398269] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16891.405946] LustreError: 523412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16892.019255] Lustre: server umount lustre-MDT0001 complete [16924.204031] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing set_hostid [16938.985801] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing load_modules_local [16988.718693] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16611 sec ========= 05:10:41 (1787389841) [16991.146654] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:10:44 (1787389844) === [16995.900138] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:10:48 (1787389848) === [17046.133960] Lustre: DEBUG MARKER: oleg243-server.virtnet: executing unload_modules_local [17049.838445] Key type lgssc unregistered [17050.379841] LNet: 532046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17050.394757] LNetError: 532046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17050.415818] LNet: Removed LNI 192.168.202.143@tcp [17051.735768] Key type .llcrypt unregistered [17051.740819] Key type ._llcrypt unregistered