[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 533189872 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003230] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008039] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009020] pid_max: default: 32768 minimum: 301 [ 0.010262] LSM: Security Framework initializing [ 0.011095] Yama: becoming mindful. [ 0.012075] SELinux: Initializing. [ 0.013137] *** VALIDATE selinux *** [ 0.024494] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028396] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029195] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032055] *** VALIDATE tmpfs *** [ 0.034168] *** VALIDATE proc *** [ 0.036067] *** VALIDATE cgroup *** [ 0.037008] *** VALIDATE cgroup2 *** [ 0.038313] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042048] Spectre V2 : User space: Vulnerable [ 0.043012] Speculative Store Bypass: Vulnerable [ 0.046000] debug: unmapping init [mem 0xffffffffaa859000-0xffffffffaa860fff] [ 0.048243] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049983] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051034] ... version: 2 [ 0.052014] ... bit width: 48 [ 0.053012] ... generic registers: 4 [ 0.054012] ... value mask: 0000ffffffffffff [ 0.055019] ... max period: 00007fffffffffff [ 0.056012] ... fixed-purpose events: 3 [ 0.056987] ... event mask: 000000070000000f [ 0.057576] rcu: Hierarchical SRCU implementation. [ 0.060229] smp: Bringing up secondary CPUs ... [ 0.061858] x86: Booting SMP configuration: [ 0.062190] .... node #0, CPUs: #1 #2 #3 [ 0.071624] smp: Brought up 1 node, 4 CPUs [ 0.073009] smpboot: Max logical packages: 1 [ 0.074010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105010] node 0 deferred pages initialised in 29ms [ 0.109217] devtmpfs: initialized [ 0.110317] x86/mm: Memory block size: 128MB [ 0.112959] gcov: version magic: 0x41383552 [ 0.115306] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.116080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.117276] pinctrl core: initialized pinctrl subsystem [ 0.118296] [ 0.118826] ************************************************************* [ 0.119017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.120013] ** ** [ 0.121011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.122014] ** ** [ 0.123012] ** This means that this kernel is built to expose internal ** [ 0.124014] ** IOMMU data structures, which may compromise security on ** [ 0.125010] ** your system. ** [ 0.126011] ** ** [ 0.127012] ** If you see this message and you are not debugging the ** [ 0.128012] ** kernel, report this immediately to your vendor! ** [ 0.129011] ** ** [ 0.130013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131011] ************************************************************* [ 0.133245] NET: Registered protocol family 16 [ 0.134611] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.135064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.136072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.138066] cpuidle: using governor menu [ 0.140184] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.142682] PCI: Using configuration type 1 for base access [ 0.145199] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.156112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.157117] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.159087] cryptd: max_cpu_qlen set to 1000 [ 0.161581] ACPI: Added _OSI(Module Device) [ 0.162010] ACPI: Added _OSI(Processor Device) [ 0.163015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.164010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.169093] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.173717] ACPI: Interpreter enabled [ 0.174089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.175017] ACPI: Using IOAPIC for interrupt routing [ 0.176162] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.177696] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.186820] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.187024] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.188014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.189086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.191618] acpiphp: Slot [2] registered [ 0.192138] acpiphp: Slot [5] registered [ 0.193109] acpiphp: Slot [6] registered [ 0.194087] acpiphp: Slot [7] registered [ 0.195108] acpiphp: Slot [8] registered [ 0.196094] acpiphp: Slot [9] registered [ 0.197210] acpiphp: Slot [10] registered [ 0.198091] acpiphp: Slot [3] registered [ 0.199119] acpiphp: Slot [4] registered [ 0.200083] acpiphp: Slot [11] registered [ 0.201085] acpiphp: Slot [12] registered [ 0.202070] acpiphp: Slot [13] registered [ 0.203179] acpiphp: Slot [14] registered [ 0.204205] acpiphp: Slot [15] registered [ 0.205112] acpiphp: Slot [16] registered [ 0.206104] acpiphp: Slot [17] registered [ 0.207123] acpiphp: Slot [18] registered [ 0.208062] acpiphp: Slot [19] registered [ 0.209061] acpiphp: Slot [20] registered [ 0.210093] acpiphp: Slot [21] registered [ 0.211202] acpiphp: Slot [22] registered [ 0.212115] acpiphp: Slot [23] registered [ 0.213077] acpiphp: Slot [24] registered [ 0.214283] acpiphp: Slot [25] registered [ 0.215249] acpiphp: Slot [26] registered [ 0.216243] acpiphp: Slot [27] registered [ 0.217078] acpiphp: Slot [28] registered [ 0.218194] acpiphp: Slot [29] registered [ 0.219216] acpiphp: Slot [30] registered [ 0.220204] acpiphp: Slot [31] registered [ 0.221077] PCI host bridge to bus 0000:00 [ 0.222038] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.223077] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.225035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.227033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.230031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.232098] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.234352] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.237280] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.241635] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.248014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.250561] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.252013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.253015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.256028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.258514] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.259450] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.261027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.262834] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.269022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.281018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.285016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.291058] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.295022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.302018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.316024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.328014] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.335021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.341019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.353018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.364395] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.370027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.378022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.394021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.403845] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.409014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.410016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.411000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.411000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.411000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.420014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.432018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.438862] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.443017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.448018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.460025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.470945] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.472351] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.474480] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.476311] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.478239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.487145] iommu: Default domain type: Passthrough [ 0.488370] SCSI subsystem initialized [ 0.489149] ACPI: bus type USB registered [ 0.490168] usbcore: registered new interface driver usbfs [ 0.492044] usbcore: registered new interface driver hub [ 0.493076] usbcore: registered new device driver usb [ 0.494287] pps_core: LinuxPPS API ver. 1 registered [ 0.496010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.498082] PTP clock support registered [ 0.499160] EDAC MC: Ver: 3.0.0 [ 0.501541] PCI: Using ACPI for IRQ routing [ 0.503928] NetLabel: Initializing [ 0.505015] NetLabel: domain hash size = 128 [ 0.506012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.508111] NetLabel: unlabeled traffic allowed by default [ 0.511189] vgaarb: loaded [ 0.513457] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.515016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.521209] clocksource: Switched to clocksource kvm-clock [ 0.647328] VFS: Disk quotas dquot_6.6.0 [ 0.649223] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.651873] *** VALIDATE ramfs *** [ 0.653232] *** VALIDATE hugetlbfs *** [ 0.655783] pnp: PnP ACPI init [ 0.658780] pnp: PnP ACPI: found 6 devices [ 0.677194] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.681208] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.683840] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.685957] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.689621] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.692198] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.696101] NET: Registered protocol family 2 [ 0.700466] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.708317] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.715575] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.720400] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.723558] TCP: Hash tables configured (established 65536 bind 65536) [ 0.726430] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.729519] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.753245] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.764986] NET: Registered protocol family 1 [ 0.767134] RPC: Registered named UNIX socket transport module. [ 0.770776] RPC: Registered udp transport module. [ 0.771942] RPC: Registered tcp transport module. [ 0.773320] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.775023] NET: Registered protocol family 44 [ 0.776617] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.778144] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.780261] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.782720] PCI: CLS 0 bytes, default 64 [ 0.810905] Unpacking initramfs... [ 3.162316] debug: unmapping init [mem 0xffff9e613cc54000-0xffff9e613ffbffff] [ 3.172212] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.175574] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.179507] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.105107] Initialise system trusted keyrings [ 4.107071] Key type blacklist registered [ 4.109539] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.123971] zbud: loaded [ 4.127978] *** VALIDATE nfs *** [ 4.129240] *** VALIDATE nfs4 *** [ 4.132274] pstore: using deflate compression [ 4.136919] Platform Keyring initialized [ 4.270590] NET: Registered protocol family 38 [ 4.271840] Key type asymmetric registered [ 4.273614] Asymmetric key parser 'x509' registered [ 4.275340] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.279076] io scheduler mq-deadline registered [ 4.281152] io scheduler kyber registered [ 4.285380] io scheduler bfq registered [ 4.287861] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.293320] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.297734] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.301733] ACPI: Power Button [PWRF] [ 4.310213] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.318901] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.341968] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.354914] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.380273] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.409470] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.439629] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.478620] Non-volatile memory driver v1.3 [ 4.480500] Linux agpgart interface v0.103 [ 4.510093] virtio_blk virtio1: [vda] 140360 512-byte logical blocks (71.9 MB/68.5 MiB) [ 4.513777] vda: detected capacity change from 0 to 71864320 [ 4.546794] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.550087] vdb: detected capacity change from 0 to 1073741824 [ 4.566131] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.568784] vdc: detected capacity change from 0 to 2621440000 [ 4.594414] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.598633] vdd: detected capacity change from 0 to 2621440000 [ 4.621321] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.625202] vde: detected capacity change from 0 to 4294967296 [ 4.642385] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.644875] vdf: detected capacity change from 0 to 4294967296 [ 4.654545] libphy: Fixed MDIO Bus: probed [ 4.661970] usbcore: registered new interface driver usbserial_generic [ 4.664364] usbserial: USB Serial support registered for generic [ 4.667198] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.671778] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.673842] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.677148] mousedev: PS/2 mouse device common for all mice [ 4.683135] rtc_cmos 00:05: RTC can wake from S4 [ 4.689072] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.688005] rtc_cmos 00:05: registered as rtc0 [ 4.697541] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.700240] intel_pstate: CPU model not supported [ 4.706724] hid: raw HID events driver (C) Jiri Kosina [ 4.709352] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.712551] usbcore: registered new interface driver usbhid [ 4.712556] usbhid: USB HID core driver [ 4.712693] drop_monitor: Initializing network drop monitor service [ 4.712840] Initializing XFRM netlink socket [ 4.714335] NET: Registered protocol family 10 [ 4.730165] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.738825] Segment Routing with IPv6 [ 4.754162] NET: Registered protocol family 17 [ 4.757650] mpls_gso: MPLS GSO support [ 4.830699] RAS: Correctable Errors collector initialized. [ 4.839563] AVX version of gcm_enc/dec engaged. [ 4.845556] AES CTR mode by8 optimization enabled [ 5.064592] sched_clock: Marking stable (5064564356, 0)->(6040953763, -976389407) [ 5.068347] registered taskstats version 1 [ 5.070615] Loading compiled-in X.509 certificates [ 5.073132] zswap: loaded using pool lzo/zbud [ 5.110388] Key type big_key registered [ 5.125760] Key type encrypted registered [ 5.127597] ima: No TPM chip found, activating TPM-bypass! [ 5.129905] ima: Allocated hash algorithm: sha1 [ 5.132268] ima: No architecture policies found [ 5.134641] evm: Initialising EVM extended attributes: [ 5.136838] evm: security.selinux [ 5.137811] evm: security.ima [ 5.138927] evm: security.capability [ 5.139990] evm: HMAC attrs: 0x1 [ 5.144415] rtc_cmos 00:05: setting system clock to 2026-04-29 00:02:15 UTC (1777420935) [ 5.152382] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 5.155613] debug: unmapping init [mem 0xffffffffaa582000-0xffffffffaa858fff] [ 5.168135] Write protecting the kernel read-only data: 28672k [ 5.173260] debug: unmapping init [mem 0xffffffffa8c03000-0xffffffffa8dfffff] [ 5.176550] debug: unmapping init [mem 0xffffffffa9514000-0xffffffffa95fffff] [ 5.242349] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.250063] systemd[1]: Detected virtualization kvm. [ 5.251772] systemd[1]: Detected architecture x86-64. [ 5.253572] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.297320] systemd[1]: No hostname configured. [ 5.298669] systemd[1]: Set hostname to . [ 5.300251] random: systemd: uninitialized urandom read (16 bytes read) [ 5.308768] systemd[1]: Initializing machine ID from random generator. [ 5.403295] random: ln: uninitialized urandom read (6 bytes read) [ 5.604142] random: systemd: uninitialized urandom read (16 bytes read) [ 5.606674] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.620594] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.637849] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.105814] device-mapper: uevent: version 1.0.3 [ 7.112599] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.584722] virtio_net virtio0 ens2: renamed from eth0 [ 8.647634] random: fast init done [ 9.008831] scsi host0: ata_piix [ 9.082879] scsi host1: ata_piix [ 9.087327] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.102435] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.660611] random: crng init done [ 14.662062] random: 7 urandom warning(s) missed due to ratelimiting [ 15.632735] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.565374] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.381387] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.238033] SELinux: Disabled at runtime. [ 20.395201] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.414544] systemd[1]: Detected virtualization kvm. [ 20.416581] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.668606] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.678015] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.698724] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.707563] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.711339] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.720975] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.731771] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ 22.024573] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.157246] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.207521] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.272624] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.739836] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.810350] EDAC sbridge: Ver: 1.1.2 [ 28.412411] Key type dns_resolver registered [ 28.769054] hrtimer: interrupt took 5037202 ns [ 29.077835] NFS: Registering the id_resolver key type [ 29.082633] Key type id_resolver registered [ 29.085717] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg114-server login: [ 72.743887] libcfs: loading out-of-tree module taints kernel. [ 72.799964] Key type ._llcrypt registered [ 72.801684] Key type .llcrypt registered [ 72.865265] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 82.624964] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 83.313717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.334886] alg: No test for adler32 (adler32-zlib) [ 84.462957] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 84.826944] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 86.479132] Key type lgssc registered [ 87.194187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.427852] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 117.045077] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 123.358043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 123.371233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 124.509524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 124.525426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 124.574843] Lustre: lustre-MDT0000: new disk, initializing [ 124.628052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 124.638247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 126.724193] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.602886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.647306] Lustre: 6480:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 133.666459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 133.671221] Lustre: Skipped 1 previous similar message [ 133.748851] Lustre: lustre-MDT0001: new disk, initializing [ 133.791101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 133.807168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 133.816598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 135.723868] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.515180] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 143.157495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 143.278457] Lustre: lustre-OST0000: new disk, initializing [ 143.281373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 143.283942] Lustre: 8380:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 143.312026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.520436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 143.527434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 143.552264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 146.416835] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.372355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 153.452626] Lustre: lustre-OST0001: new disk, initializing [ 153.454927] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 153.458290] Lustre: 9437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 153.487702] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 155.799208] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.269278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 160.275334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 160.314547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 164.031728] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 169.508595] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 177.201251] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing check_logdir /tmp/testlogs/ [ 179.330524] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing yml_node [ 182.011071] Lustre: DEBUG MARKER: Client: 2.17.50.194 [ 183.560769] Lustre: DEBUG MARKER: MDS: 2.17.50.194 [ 185.051688] Lustre: DEBUG MARKER: OSS: 2.17.50.194 [ 186.094881] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 28 20:05:15 EDT 2026 [ 195.717301] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 196.454641] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 210.915704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 210.925181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 210.937967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 211.428794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 211.443589] Lustre: Skipped 2 previous similar messages [ 211.452464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 211.457061] Lustre: Skipped 3 previous similar messages [ 215.504804] Lustre: server umount lustre-MDT0000 complete [ 217.547796] LustreError: 7410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777421147 with bad export cookie 2935358283049690427 [ 217.552181] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 217.554206] LustreError: 7410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 221.663575] LustreError: 10408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 221.665041] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 221.673643] LustreError: 10408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 221.684945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 221.694134] Lustre: Skipped 1 previous similar message [ 222.688952] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 222.697649] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 223.807234] Lustre: server umount lustre-MDT0001 complete [ 226.372920] Lustre: server umount lustre-OST0000 complete [ 228.884782] Lustre: server umount lustre-OST0001 complete [ 232.429801] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 237.810702] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 258.499800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.673330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 258.699578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 258.760981] Lustre: lustre-MDT0000: new disk, initializing [ 258.822554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.833875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 261.080563] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 267.211545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 267.304494] Lustre: 14944:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 267.357856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 267.367768] Lustre: Skipped 1 previous similar message [ 267.411609] Lustre: lustre-MDT0001: new disk, initializing [ 267.466257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.487962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 267.497387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 269.605784] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 274.228074] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 276.131656] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 279.892462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 280.058640] Lustre: 16684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 281.738505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 281.792661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 282.976600] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 286.835413] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 288.756767] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 288.882805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 290.928734] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 291.039535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 292.327356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 292.327632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.331804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 292.346872] Lustre: Skipped 2 previous similar messages [ 292.352363] Lustre: Skipped 3 previous similar messages [ 297.444663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 297.448075] Lustre: Skipped 1 previous similar message [ 298.075060] Lustre: server umount lustre-OST0000 complete [ 302.560796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.567271] Lustre: Skipped 1 previous similar message [ 307.680279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.685507] Lustre: Skipped 3 previous similar messages [ 314.336043] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 314.419752] Lustre: server umount lustre-MDT0000 complete [ 316.252826] LustreError: 15730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777421246 with bad export cookie 2935358283049691967 [ 316.257441] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.425667] Lustre: server umount lustre-MDT0001 complete [ 318.266830] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:07:27 (1777421247) [ 319.310794] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 320.178488] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:07:29 (1777421249) [ 323.206068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.448269] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.478752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.481367] Lustre: Skipped 1 previous similar message [ 325.121948] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 328.672725] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.309969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 331.377736] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 333.990036] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 335.531178] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 338.555189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 340.688381] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 342.917335] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 358.881950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 358.883465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.885731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 358.890231] Lustre: Skipped 1 previous similar message [ 358.894072] Lustre: Skipped 3 previous similar messages [ 363.488218] Lustre: server umount lustre-OST0000 complete [ 365.535947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 371.367338] Lustre: server umount lustre-MDT0000 complete [ 372.983356] LustreError: 21411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777421303 with bad export cookie 2935358283049692849 [ 372.985596] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.987271] LustreError: 21411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 373.127175] Lustre: server umount lustre-MDT0001 complete [ 377.500598] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 378.829647] Key type lgssc unregistered [ 378.988239] LNet: 22145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.991654] LNetError: 22145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 380.007705] LNet: Removed LNI 192.168.201.114@tcp [ 380.449396] Key type .llcrypt unregistered [ 380.451496] Key type ._llcrypt unregistered [ 392.001293] Key type ._llcrypt registered [ 392.003023] Key type .llcrypt registered [ 392.069220] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 392.584191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 392.594938] alg: No test for adler32 (adler32-zlib) [ 393.473701] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 393.608140] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 395.239215] Key type lgssc registered [ 395.687485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 399.413308] loop: module loaded [ 399.434086] loop0: detected capacity change from 0 to 204800000 [ 399.533522] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 399.537919] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 399.686401] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 402.467722] loop0: detected capacity change from 0 to 204800000 [ 402.582419] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 402.588778] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 402.598692] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 404.909135] loop0: detected capacity change from 0 to 204800000 [ 407.342342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 407.350647] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.464458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 408.499963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 408.559260] Lustre: lustre-MDT0000: new disk, initializing [ 408.606640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.619114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 410.670504] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 415.496364] loop1: detected capacity change from 0 to 204800000 [ 418.197554] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.359710] Lustre: lustre-OST07c6: new disk, initializing [ 418.366244] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 418.371613] Lustre: Skipped 1 previous similar message [ 418.374585] Lustre: 25269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 418.407618] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 421.251642] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 423.939980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 423.944283] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 423.958158] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 430.380710] Lustre: 24276:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 430.384594] Lustre: 24276:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 430.388089] Lustre: 24276:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 430.391784] Lustre: 24276:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 430.395873] Lustre: 24276:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 430.399139] Lustre: 24276:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 450.039136] Lustre: 24275:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 450.042889] Lustre: 24275:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 450.046262] Lustre: 24275:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 450.049711] Lustre: 24275:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 450.052606] Lustre: 24275:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 450.055294] Lustre: 24275:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 453.693232] Lustre: 24275:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 453.696224] Lustre: 24275:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 453.698604] Lustre: 24275:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 453.701013] Lustre: 24275:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 453.703433] Lustre: 24275:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 453.705944] Lustre: 24275:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 453.708531] Lustre: 24275:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 453.711574] Lustre: 24275:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 453.714345] Lustre: 24275:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 453.717732] Lustre: 24275:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 453.720711] Lustre: 24275:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 453.723704] Lustre: 24275:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.783554] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 456.787223] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 456.790024] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 456.792747] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.795714] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 456.798411] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.801093] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 456.806060] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.808579] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 456.811280] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.813968] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 456.816313] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 463.402825] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 463.407033] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 3 previous similar messages [ 463.409670] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 463.411997] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 463.414478] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 463.417975] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 463.420787] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 463.423832] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 463.426579] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 463.429297] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 463.432490] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 463.435538] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 473.135859] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 473.139637] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 473.141712] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 473.144567] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 473.147344] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 473.149976] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 473.152802] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 473.156268] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 473.159026] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 473.162130] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 473.164676] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 473.167368] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 516.064580] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.067305] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.073593] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 520.407901] Lustre: server umount lustre-OST07c6 complete [ 524.251900] Lustre: server umount lustre-MDT0000 complete [ 530.938676] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 532.130910] Key type lgssc unregistered [ 532.255864] LNet: 27814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.259848] LNetError: 27814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.268317] LNet: Removed LNI 192.168.201.114@tcp [ 532.623315] Key type .llcrypt unregistered [ 532.624959] Key type ._llcrypt unregistered [ 540.620153] Key type ._llcrypt registered [ 540.621649] Key type .llcrypt registered [ 540.663982] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 541.084635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 541.112369] alg: No test for adler32 (adler32-zlib) [ 541.997461] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 542.092914] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 543.687188] Key type lgssc registered [ 544.130333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 547.202040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 547.210508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.395399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.888070] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 553.095519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.239841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 554.675119] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 556.655153] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 558.031410] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 560.488835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 560.606562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 562.654420] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 565.011446] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 572.908088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 583.139142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.139456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.151305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 583.157835] Lustre: Skipped 1 previous similar message [ 588.258317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 588.262039] Lustre: Skipped 1 previous similar message [ 588.463113] Lustre: server umount lustre-OST0000 complete [ 593.377118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.377709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 593.381955] Lustre: Skipped 2 previous similar messages [ 596.667505] Lustre: server umount lustre-MDT0000 complete [ 598.430193] LustreError: 29028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777421528 with bad export cookie 9353009885681714945 [ 598.433861] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.436191] LustreError: 29028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 598.597975] Lustre: server umount lustre-MDT0001 complete [ 603.237751] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 604.517798] Key type lgssc unregistered [ 604.652573] LNet: 32446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.656668] LNetError: 32446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.669505] LNet: Removed LNI 192.168.201.114@tcp [ 605.023416] Key type .llcrypt unregistered [ 605.025046] Key type ._llcrypt unregistered [ 613.897985] Key type ._llcrypt registered [ 613.899135] Key type .llcrypt registered [ 613.946954] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 614.293978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 614.313579] alg: No test for adler32 (adler32-zlib) [ 615.221211] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 615.334247] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 616.935172] Key type lgssc registered [ 617.367572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 620.537443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 620.546421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.761256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 623.220695] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 626.465272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.613756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 628.119978] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 630.177079] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 631.396828] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 633.944449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 634.073414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.079746] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 638.117527] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 647.408682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 656.864073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 656.867597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.874124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 657.888872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 657.889362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 657.889917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.898732] Lustre: Skipped 1 previous similar message [ 660.511971] Lustre: server umount lustre-OST0000 complete [ 662.495900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 662.499291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.504949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 668.128456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 668.130791] Lustre: Skipped 3 previous similar messages [ 673.248636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 673.250961] Lustre: Skipped 1 previous similar message [ 676.319516] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 676.391968] Lustre: server umount lustre-MDT0000 complete [ 677.982515] LustreError: 33657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777421608 with bad export cookie 2999850475939618199 [ 677.986080] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.987392] LustreError: 33657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 678.138897] Lustre: server umount lustre-MDT0001 complete [ 682.250471] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 683.380275] Key type lgssc unregistered [ 683.524617] LNet: 36883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.527754] LNetError: 36883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.535456] LNet: Removed LNI 192.168.201.114@tcp [ 683.888253] Key type .llcrypt unregistered [ 683.889775] Key type ._llcrypt unregistered [ 694.038420] Key type ._llcrypt registered [ 694.039983] Key type .llcrypt registered [ 694.085429] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 694.427084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.447064] alg: No test for adler32 (adler32-zlib) [ 695.327629] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 695.417231] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 697.007108] Key type lgssc registered [ 697.427794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 700.351922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 700.365658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.606531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.203597] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 706.650870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.829743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 708.585723] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 710.899536] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 712.371684] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 715.366672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 715.517251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.737317] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 719.680129] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 725.734755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 732.640898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 732.645098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.650259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.446359] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 736.228846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.233399] Lustre: Skipped 1 previous similar message [ 738.843794] Lustre: server umount lustre-MDT0000 complete [ 740.410808] LustreError: 38240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777421670 with bad export cookie 16109844473274017548 [ 740.410808] LustreError: 39873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777421670 with bad export cookie 16109844473274017548 [ 740.411462] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 740.420984] LustreError: 39873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 740.556709] LustreError: 38256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.564606] LustreError: 38256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 740.565649] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 740.570967] Lustre: Skipped 3 previous similar messages [ 741.343972] LustreError: 38254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.344448] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.375826] LustreError: 38254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 745.376123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 745.384399] Lustre: Skipped 1 previous similar message [ 746.608129] Lustre: server umount lustre-MDT0001 complete [ 764.694673] Lustre: server umount lustre-OST0000 complete [ 770.783105] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 771.919493] Key type lgssc unregistered [ 772.056391] LNet: 41781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.061257] LNetError: 41781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.074324] LNet: Removed LNI 192.168.201.114@tcp [ 772.402218] Key type .llcrypt unregistered [ 772.403548] Key type ._llcrypt unregistered [ 781.908809] Key type ._llcrypt registered [ 781.910396] Key type .llcrypt registered [ 781.963532] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 782.342713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 782.387875] alg: No test for adler32 (adler32-zlib) [ 783.252501] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 783.344029] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 784.943137] Key type lgssc registered [ 785.351118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 788.502443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 788.512052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.696320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.127527] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 794.401027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.537248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 795.988681] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 797.988304] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 799.360673] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 802.136400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 802.274855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.376355] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 806.333215] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 815.589478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 820.705205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.705923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 820.706358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 820.709977] Lustre: Skipped 1 previous similar message [ 822.997262] Lustre: lustre-OST0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 823.000579] Lustre: Skipped 1 previous similar message [ 825.824888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 825.827644] Lustre: Skipped 1 previous similar message [ 826.517484] Lustre: server umount lustre-OST0000 complete [ 840.672035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 840.674692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.680447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 845.721201] Lustre: server umount lustre-MDT0000 complete [ 846.304161] LustreError: 43158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.310936] LustreError: 43158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 847.009451] LustreError: 43139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777421777 with bad export cookie 2615437139197069260 [ 847.012751] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 847.014492] LustreError: 43139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 853.169837] Lustre: server umount lustre-MDT0001 complete [ 857.257853] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 858.768194] Key type lgssc unregistered [ 858.952384] LNet: 46533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.964239] LNetError: 46533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.980116] LNet: Removed LNI 192.168.201.114@tcp [ 859.484623] Key type .llcrypt unregistered [ 859.486895] Key type ._llcrypt unregistered [ 879.786668] Key type ._llcrypt registered [ 879.789365] Key type .llcrypt registered [ 879.887758] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 880.724228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 880.929390] alg: No test for adler32 (adler32-zlib) [ 881.920267] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 882.053590] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 883.711167] Key type lgssc registered [ 884.583111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 891.730366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 891.740875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.160715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.437095] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 903.374184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 903.629569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.633345] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 913.693914] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 917.816414] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 924.576351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 924.802188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.006041] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 934.311859] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 940.146604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 948.072418] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 950.261723] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 961.714125] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-2820h18m10s [ 962.774441] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 965.090446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 965.101423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.115106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.182441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.187813] Lustre: Skipped 3 previous similar messages [ 971.409869] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 971.416617] Lustre: Skipped 3 previous similar messages [ 977.622972] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 977.635917] Lustre: Skipped 3 previous similar messages [ 978.399195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 978.552185] Lustre: server umount lustre-MDT0000 complete [ 980.861110] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 983.757370] LustreError: 47944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 983.771079] LustreError: 47944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 984.549308] LustreError: 50052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.562533] LustreError: 50052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 985.411345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.500274] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 985.678627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.724480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 986.741567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 988.251169] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 990.510276] Lustre: DEBUG MARKER: Wait for df (36045) ... [ 991.503277] Lustre: DEBUG MARKER: done [ 994.786995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 994.797601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.815677] Lustre: Skipped 2 previous similar messages [ 994.822357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.825222] Lustre: Skipped 3 previous similar messages [ 995.814078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 995.822049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.143434] Lustre: server umount lustre-OST0000 complete [ 1007.583992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1007.586633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.598216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.604522] Lustre: Skipped 3 previous similar messages [ 1009.525086] Lustre: server umount lustre-MDT0000 complete [ 1012.408418] LustreError: 47928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777421942 with bad export cookie 17375505838682620730 [ 1012.415850] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.419868] LustreError: 47928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1012.642082] Lustre: server umount lustre-MDT0001 complete [ 1019.798853] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1021.558251] Key type lgssc unregistered [ 1021.729500] LNet: 52522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.733235] LNetError: 52522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.742752] LNet: Removed LNI 192.168.201.114@tcp [ 1022.184445] Key type .llcrypt unregistered [ 1022.187404] Key type ._llcrypt unregistered [ 1046.135832] Key type ._llcrypt registered [ 1046.138471] Key type .llcrypt registered [ 1046.241827] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1047.074511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1047.091507] alg: No test for adler32 (adler32-zlib) [ 1048.001476] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1048.137146] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1049.767764] Key type lgssc registered [ 1050.516740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1055.548808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1055.563892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.736954] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1056.763993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1056.901179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1059.332767] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1065.196919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.266835] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1065.293092] Lustre: 54691:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1065.476536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1068.766323] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1073.324655] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1076.103645] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1080.727269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1080.869983] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1081.021536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.906588] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1087.089146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1088.334292] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1094.190401] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-2820h20m23s [ 1095.448870] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1101.252950] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1103.215621] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1103.398409] LustreError: 54697:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1123.581809] Lustre: lustre-MDT0000: Client 0363578c-b796-4009-ae51-6ac4d36dea29 (at 192.168.201.14@tcp) reconnecting [ 1143.415192] LustreError: 54697:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1145.172701] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1147.098971] Lustre: DEBUG MARKER: done [ 1153.506293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1153.512664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.521520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.029142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1154.044413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.046418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.075096] Lustre: Skipped 1 previous similar message [ 1157.061587] Lustre: server umount lustre-OST0000 complete [ 1162.722081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1162.730542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.756981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.856644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.867163] Lustre: Skipped 4 previous similar messages [ 1166.580632] Lustre: server umount lustre-MDT0000 complete [ 1169.604898] LustreError: 54683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422099 with bad export cookie 15539458266651340940 [ 1169.612555] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.618885] LustreError: 54683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1169.881874] Lustre: server umount lustre-MDT0001 complete [ 1178.268904] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1180.312826] Key type lgssc unregistered [ 1180.542587] LNet: 58660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.547495] LNetError: 58660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.557553] LNet: Removed LNI 192.168.201.114@tcp [ 1181.206159] Key type .llcrypt unregistered [ 1181.207547] Key type ._llcrypt unregistered [ 1210.544084] Key type ._llcrypt registered [ 1210.545872] Key type .llcrypt registered [ 1210.606849] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1211.176889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1211.250836] alg: No test for adler32 (adler32-zlib) [ 1212.148922] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1212.254612] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1213.855160] Key type lgssc registered [ 1214.365293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1215.830791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1215.840386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.968521] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1216.980239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1217.051153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1217.488461] Lustre: Failing over lustre-MDT0000 [ 1217.607134] Lustre: server umount lustre-MDT0000 complete [ 1220.515835] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:22:30 (1777422150) [ 1223.699843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.936852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.910793] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1230.068559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.119967] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1230.136671] Lustre: 61421:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1230.290115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1232.076347] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1234.778736] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1236.780064] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1240.235439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1240.331406] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1240.445557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.409360] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1246.454192] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1246.777616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1250.836562] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1255.906086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1255.909382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.917761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.932609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.933421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.935475] Lustre: Skipped 1 previous similar message [ 1256.941525] Lustre: Skipped 1 previous similar message [ 1259.241421] Lustre: server umount lustre-MDT0000 complete [ 1260.850382] LustreError: 62141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422191 with bad export cookie 16659493103469779794 [ 1260.857140] LustreError: 62141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1260.857242] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.992612] Lustre: server umount lustre-MDT0001 complete [ 1262.581522] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1264.393525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1268.140230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.199792] LustreError: 60342:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000068108155 x1863762200002560/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1271.401104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.443664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1273.005373] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1276.563161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.674424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.727155] Lustre: 60345:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777422192/real 1777422192] req@000000003f1d532c x1863762200002176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777422208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1277.735745] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1278.431835] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1280.685956] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1280.800081] Lustre: 60344:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777422195/real 1777422195] req@00000000267ec524 x1863762200002304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777422211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1282.355220] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1284.064432] Lustre: 60344:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777422197/real 1777422197] req@00000000465c18d7 x1863762200002432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777422213 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1287.136417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1287.140133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.149668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1287.631129] Lustre: 60346:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777422201/real 1777422201] req@000000004b4f90a7 x1863762200002688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777422217 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1290.719517] Lustre: server umount lustre-MDT0000 complete [ 1292.201929] LustreError: 61415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422222 with bad export cookie 16659493103469782055 [ 1292.205799] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1292.207816] LustreError: 61415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1292.344970] Lustre: server umount lustre-MDT0001 complete [ 1295.181063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1298.945263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1302.432049] LustreError: 60342:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ed8cb47f x1863762200022400/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1302.589148] LustreError: 65563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.596740] LustreError: 65563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1302.630474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.633511] Lustre: Skipped 1 previous similar message [ 1302.655958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1304.319917] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1307.746499] LustreError: 65563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.746717] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.748562] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1307.748568] Lustre: Skipped 1 previous similar message [ 1307.759626] Lustre: Skipped 2 previous similar messages [ 1307.796453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.959152] Lustre: 60345:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777422223/real 1777422223] req@0000000067ffa141 x1863762200022016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777422239 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1309.574260] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1311.672386] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1312.915312] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1313.254090] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1319.391147] Lustre: 60343:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777422232/real 1777422232] req@0000000051e86638 x1863762200022528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777422248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1319.400230] Lustre: 60343:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1323.490013] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.491370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1323.493879] Lustre: Skipped 2 previous similar messages [ 1323.494154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1323.496285] Lustre: Skipped 5 previous similar messages [ 1326.108708] Lustre: server umount lustre-MDT0000 complete [ 1327.420199] LustreError: 61414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422257 with bad export cookie 16659493103469783434 [ 1327.423157] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.425384] LustreError: 61414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1327.554353] Lustre: server umount lustre-MDT0001 complete [ 1329.989680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1331.181575] Lustre: DEBUG MARKER: files compared the same [ 1342.188938] Lustre: server umount lustre-OST0000 complete [ 1348.206799] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1349.344179] Key type lgssc unregistered [ 1349.503491] LNet: 70334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.507886] LNetError: 70334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.565492] LNet: Removed LNI 192.168.201.114@tcp [ 1350.942377] Key type .llcrypt unregistered [ 1350.944285] Key type ._llcrypt unregistered [ 1360.879108] Key type ._llcrypt registered [ 1360.880726] Key type .llcrypt registered [ 1360.922293] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1361.260898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.268377] alg: No test for adler32 (adler32-zlib) [ 1362.133589] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1362.226210] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1363.807170] Key type lgssc registered [ 1364.200245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1367.107070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1367.114683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.313597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.870736] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug +malloc all [ 1373.122425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.233579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.687140] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug +malloc all [ 1376.846065] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1378.291703] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1380.955662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1381.068209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.324894] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug +malloc all [ 1385.496289] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1394.403852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1399.775692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1399.778087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.779775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1399.784104] Lustre: Skipped 1 previous similar message [ 1400.917615] Lustre: server umount lustre-OST0000 complete [ 1403.871669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1403.875396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.879361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1403.881205] Lustre: Skipped 1 previous similar message [ 1404.577127] Lustre: server umount lustre-MDT0000 complete [ 1404.899870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.903764] LustreError: 72947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.918628] LustreError: 72947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1406.144752] LustreError: 71694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422336 with bad export cookie 7170939815423651774 [ 1406.146567] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1406.149054] LustreError: 71694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1406.261468] Lustre: server umount lustre-MDT0001 complete [ 1410.452988] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1411.525073] Key type lgssc unregistered [ 1411.666440] LNet: 74908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.669706] LNetError: 74908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.680550] LNet: Removed LNI 192.168.201.114@tcp [ 1411.993388] Key type .llcrypt unregistered [ 1411.994922] Key type ._llcrypt unregistered [ 1422.158142] Key type ._llcrypt registered [ 1422.159588] Key type .llcrypt registered [ 1422.198980] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1422.556131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1422.607211] alg: No test for adler32 (adler32-zlib) [ 1423.468070] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1423.558197] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1425.143170] Key type lgssc registered [ 1425.557914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1428.461376] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1428.467887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1443.935448] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1449.055299] LustreError: 76259:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.114@tcp: failed processing log, type 4: rc = -110 [ 1475.679207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.784844] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1479.779241] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1482.612138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.657874] Lustre: *** cfs_fail_loc=706, val=0*** [ 1482.798732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.823487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1484.143806] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1487.141837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.268079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1488.572252] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1490.617894] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1491.894511] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1492.959955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1492.963111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.967995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1495.523323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1495.527463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.777213] Lustre: server umount lustre-OST0000 complete [ 1500.640170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.640685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1500.645833] Lustre: Skipped 1 previous similar message [ 1500.648527] Lustre: Skipped 1 previous similar message [ 1505.760052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1505.762451] Lustre: Skipped 1 previous similar message [ 1506.273421] Lustre: server umount lustre-MDT0000 complete [ 1507.642325] LustreError: 76265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422437 with bad export cookie 10216593473646527026 [ 1507.642947] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.647044] LustreError: 76265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1507.766757] Lustre: server umount lustre-MDT0001 complete [ 1511.366591] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1512.400712] Key type lgssc unregistered [ 1512.526340] LNet: 79519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.529179] LNetError: 79519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.539278] LNet: Removed LNI 192.168.201.114@tcp [ 1512.836341] Key type .llcrypt unregistered [ 1512.837252] Key type ._llcrypt unregistered [ 1522.931692] Key type ._llcrypt registered [ 1522.933325] Key type .llcrypt registered [ 1522.975652] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1523.443834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1523.511727] alg: No test for adler32 (adler32-zlib) [ 1524.413674] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1524.513933] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1526.103172] Key type lgssc registered [ 1526.562834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1530.059081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1530.065673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.667748] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1535.613723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.814878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1537.334840] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1540.593529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1540.751224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.822778] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1545.889189] LustreError: 81569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1545.896287] LustreError: 81569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1546.077370] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1546.083048] Lustre: MGS: Client 911d8cfc-0073-4ff7-90af-d418a0f42c1c (at 0@lo) reconnecting [ 1546.186874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1546.233421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1547.674629] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1556.448900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.451875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1556.452363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1556.453809] Lustre: Skipped 1 previous similar message [ 1561.568972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1561.571172] Lustre: Skipped 2 previous similar messages [ 1561.679868] Lustre: server umount lustre-OST0000 complete [ 1566.689696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.689809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1566.695089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.700193] Lustre: Skipped 1 previous similar message [ 1569.093746] Lustre: server umount lustre-MDT0000 complete [ 1570.664532] Lustre: server umount lustre-MDT0001 complete [ 1572.179281] Lustre: server umount MGS complete [ 1576.772132] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:28:26 (1777422506) [ 1587.207608] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 1590.401469] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1603.363456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.765160] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1608.899183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.005168] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1609.028526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1609.068873] Lustre: lustre-MDT0001: new disk, initializing [ 1609.104251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.115959] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1609.119537] Lustre: 88625:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1609.123578] LustreError: 88625:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1610.466327] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1614.802921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1614.915772] Lustre: lustre-OST0000: new disk, initializing [ 1614.917596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1614.920625] Lustre: 89375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1614.946416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.995591] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1619.684647] Lustre: 89814:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000a25df63c x1863762527159680/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777422566 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1620.322943] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1622.837372] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1622.844224] Lustre: MGS: Client 37d1aa00-10b2-4a0f-ba72-45b422413f25 (at 0@lo) reconnecting [ 1622.859932] Lustre: 90307:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1622.868162] LustreError: 90307:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1622.870686] LustreError: 90307:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1622.880213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1622.915212] Lustre: lustre-MDT0000: new disk, initializing [ 1622.965484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.978845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1624.031858] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1624.032070] Lustre: 90323:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000969e1066 x1863762527165184/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777422570 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1624.043661] Lustre: 90323:0:(client.c:1637:after_reply()) Skipped 1 previous similar message [ 1624.390991] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1628.770210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1628.773545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1629.092469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1649.632161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1649.632353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.635792] LustreError: Skipped 1 previous similar message [ 1649.636383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1649.640308] Lustre: Skipped 2 previous similar messages [ 1654.616121] Lustre: server umount lustre-OST0000 complete [ 1657.824576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1657.824669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.829149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1657.833759] Lustre: Skipped 1 previous similar message [ 1657.835909] Lustre: Skipped 3 previous similar messages [ 1662.223821] Lustre: server umount lustre-MDT0000 complete [ 1662.944052] LustreError: 89815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.951648] LustreError: 89815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1669.734073] Lustre: server umount lustre-MDT0001 complete [ 1670.939444] Lustre: server umount MGS complete [ 1674.841344] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:30:04 (1777422604) [ 1681.251355] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1682.307842] Key type lgssc unregistered [ 1682.445734] LNet: 92796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.448645] LNetError: 92796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.459729] LNet: Removed LNI 192.168.201.114@tcp [ 1682.767670] Key type .llcrypt unregistered [ 1682.768962] Key type ._llcrypt unregistered [ 1689.975320] Key type ._llcrypt registered [ 1689.976739] Key type .llcrypt registered [ 1690.015976] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1690.410931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1690.429976] alg: No test for adler32 (adler32-zlib) [ 1691.298287] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1691.389672] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1692.975155] Key type lgssc registered [ 1693.340544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1695.937408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1695.949072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.121461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.418413] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1701.188448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1701.314357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1702.556423] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1704.367091] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1705.607062] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1708.048221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1708.202440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.351641] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1712.400170] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1722.533110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1726.944627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1726.947037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.951479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1727.970477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1727.970545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1727.971648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.973783] Lustre: Skipped 1 previous similar message [ 1731.666652] Lustre: server umount lustre-OST0000 complete [ 1733.088880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.089507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1733.093334] Lustre: Skipped 1 previous similar message [ 1738.208410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.211148] Lustre: Skipped 2 previous similar messages [ 1739.031952] Lustre: server umount lustre-MDT0000 complete [ 1740.302363] LustreError: 96445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422670 with bad export cookie 948474144083746885 [ 1740.305095] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.307165] LustreError: 96445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1740.427611] Lustre: server umount lustre-MDT0001 complete [ 1744.102953] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1745.284889] Key type lgssc unregistered [ 1745.427599] LNet: 97178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.431521] LNetError: 97178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.440365] LNet: Removed LNI 192.168.201.114@tcp [ 1745.753282] Key type .llcrypt unregistered [ 1745.754353] Key type ._llcrypt unregistered [ 1752.385406] Key type ._llcrypt registered [ 1752.386480] Key type .llcrypt registered [ 1752.427476] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1752.821849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1752.857943] alg: No test for adler32 (adler32-zlib) [ 1753.719150] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1753.802829] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1755.383157] Key type lgssc registered [ 1755.766224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1757.911299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1757.911304] LustreError: 98222:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1757.917096] Lustre: Skipped 1 previous similar message [ 1757.918384] LustreError: 98223:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1757.921903] LustreError: 98222:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 1757.924607] LustreError: 98222:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1757.928189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.929140] LustreError: 98222:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1759.091580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.956207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.073081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1763.547991] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1765.450074] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing lsmod [ 1766.520539] LustreError: 99392:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1766.523464] LustreError: 99394:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1766.527487] LustreError: 99392:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1766.530208] LustreError: 99392:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1766.533673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1766.534149] LustreError: 99392:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1766.673971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1771.810718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.812550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1771.817319] Lustre: Skipped 1 previous similar message [ 1773.848550] Lustre: server umount lustre-MDT0000 complete [ 1775.218122] LustreError: 99584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422705 with bad export cookie 2116769193506153702 [ 1775.220726] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1775.222235] LustreError: 99584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1775.903314] LustreError: 98692:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007b6af86e x1863762768313088/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1775.958560] Lustre: server umount lustre-MDT0001 complete [ 1787.564503] Lustre: server umount lustre-OST0000 complete [ 1790.789857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.990856] LustreError: 100462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.998996] LustreError: 100462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1791.022681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.521947] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1795.594891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.728901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1797.100473] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1798.994400] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1800.332225] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1802.931271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1805.267525] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1807.225050] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1816.360502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1821.667501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.667548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1821.668697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1821.668704] Lustre: Skipped 2 previous similar messages [ 1821.672368] Lustre: Skipped 2 previous similar messages [ 1821.675503] LustreError: Skipped 1 previous similar message [ 1825.298896] Lustre: server umount lustre-OST0000 complete [ 1831.905283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1831.905303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1831.913063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.915370] Lustre: Skipped 2 previous similar messages [ 1837.023971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1837.026159] Lustre: Skipped 1 previous similar message [ 1841.119154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1841.190874] Lustre: server umount lustre-MDT0000 complete [ 1842.143946] LustreError: 100462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1842.150117] LustreError: 100462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1842.491497] LustreError: 100441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422772 with bad export cookie 2116769193506154374 [ 1842.494572] LustreError: 100441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1842.495153] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1842.606429] Lustre: server umount lustre-MDT0001 complete [ 1846.398661] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1847.544853] Key type lgssc unregistered [ 1847.690527] LNet: 103658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.693293] LNetError: 103658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.704332] LNet: Removed LNI 192.168.201.114@tcp [ 1848.037562] Key type .llcrypt unregistered [ 1848.038714] Key type ._llcrypt unregistered [ 1857.986834] Key type ._llcrypt registered [ 1857.988033] Key type .llcrypt registered [ 1858.036421] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1858.474419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1858.561693] alg: No test for adler32 (adler32-zlib) [ 1859.442800] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1859.551721] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1861.143225] Key type lgssc registered [ 1861.596612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1864.608797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1864.622571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.801621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.415922] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1870.506602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.659551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1871.955038] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1873.719839] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1874.831912] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1877.089669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.201892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1878.999885] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1880.757962] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1891.557609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1898.578379] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1899.383434] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1901.023917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.026578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.031630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.048981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1902.050591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.052476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.511453] Lustre: server umount lustre-OST0000 complete [ 1911.775923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1911.779791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.784665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.786237] Lustre: Skipped 1 previous similar message [ 1917.408313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.410952] Lustre: Skipped 3 previous similar messages [ 1922.015111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1922.071122] Lustre: server umount lustre-MDT0000 complete [ 1922.528347] LustreError: 107401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.533200] LustreError: 107401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1923.291335] LustreError: 106650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422853 with bad export cookie 3044736736712277050 [ 1923.293924] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1923.294281] LustreError: 106650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1923.398165] Lustre: server umount lustre-MDT0001 complete [ 1926.544387] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1927.471540] Key type lgssc unregistered [ 1927.582293] LNet: 108341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.584460] LNetError: 108341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.591323] LNet: Removed LNI 192.168.201.114@tcp [ 1927.875855] Key type .llcrypt unregistered [ 1927.876819] Key type ._llcrypt unregistered [ 1934.304242] Key type ._llcrypt registered [ 1934.305703] Key type .llcrypt registered [ 1934.340936] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1934.744399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1934.752937] alg: No test for adler32 (adler32-zlib) [ 1935.610415] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 1935.696629] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1937.279167] Key type lgssc registered [ 1937.615949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1939.962379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1939.967946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.106717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.359259] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1945.077662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.206983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1946.431663] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1948.251286] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1949.431116] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1951.750991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1951.873179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.736208] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1955.651343] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1967.270690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1975.776110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1975.781544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1975.787975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.825832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1977.826524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.827245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.837584] Lustre: Skipped 1 previous similar message [ 1980.118897] Lustre: server umount lustre-OST0000 complete [ 1982.944662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.945351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.949437] Lustre: Skipped 1 previous similar message [ 1987.671344] Lustre: server umount lustre-MDT0000 complete [ 1988.064889] LustreError: 110233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.071301] LustreError: 110233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1988.953236] LustreError: 111987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777422919 with bad export cookie 3008482857955220252 [ 1988.955817] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1988.957951] LustreError: 111987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1989.071362] Lustre: server umount lustre-MDT0001 complete [ 1992.746984] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1993.729787] Key type lgssc unregistered [ 1993.859386] LNet: 112720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.862086] LNetError: 112720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.868309] LNet: Removed LNI 192.168.201.114@tcp [ 1994.139831] Key type .llcrypt unregistered [ 1994.140854] Key type ._llcrypt unregistered [ 2002.678478] Key type ._llcrypt registered [ 2002.679988] Key type .llcrypt registered [ 2002.723859] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2003.091094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2003.120337] alg: No test for adler32 (adler32-zlib) [ 2003.961108] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 2004.037284] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2005.615224] Key type lgssc registered [ 2005.960934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2008.560602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2008.568938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.760177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.019396] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2013.986742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.108594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2015.503652] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2017.301921] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2018.389708] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2020.448190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.547166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.290156] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2023.939471] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2034.917190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2041.858074] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2043.029666] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2043.032401] Lustre: Skipped 2 previous similar messages [ 2044.367565] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2044.369949] Lustre: Skipped 2 previous similar messages [ 2048.512962] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2048.524979] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2053.894739] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.14@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2053.898760] Lustre: Skipped 2 previous similar messages [ 2057.976330] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.14@tcp 192.168.0.[2,10]@tcp [ 2063.249238] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.14/24@tcp in log lustre-MDT0000 [ 2063.252187] Lustre: Skipped 2 previous similar messages [ 2064.118228] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.14/24@tcp [ 2064.120928] Lustre: Skipped 1 previous similar message [ 2069.983991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2069.987143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.991668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.009927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.010615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2071.010658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.012747] Lustre: Skipped 1 previous similar message [ 2074.130907] Lustre: server umount lustre-OST0000 complete [ 2075.615872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2075.618329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.623066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.461603] Lustre: server umount lustre-MDT0000 complete [ 2078.852242] LustreError: 116411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423009 with bad export cookie 8702550548532493689 [ 2078.854603] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2078.856612] LustreError: 116411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2081.004318] Lustre: server umount lustre-MDT0001 complete [ 2084.497547] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2085.533242] Key type lgssc unregistered [ 2085.671378] LNet: 118699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.674119] LNetError: 118699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.680356] LNet: Removed LNI 192.168.201.114@tcp [ 2085.982579] Key type .llcrypt unregistered [ 2085.984271] Key type ._llcrypt unregistered [ 2096.121759] Key type ._llcrypt registered [ 2096.122845] Key type .llcrypt registered [ 2096.158831] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2096.489913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.530040] alg: No test for adler32 (adler32-zlib) [ 2097.385015] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 2097.465848] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2099.047144] Key type lgssc registered [ 2099.431107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2101.920589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2101.927322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.077600] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2103.082060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.086505] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.14/24@tcp [ 2104.327570] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2107.148789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.248950] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2107.257969] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.14/24@tcp [ 2107.273942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2108.521467] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2110.258475] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2111.486382] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2113.757564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2113.870638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.767464] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2117.480214] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2128.231503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2137.567756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2137.570493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.574270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2138.593013] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.593921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2138.594774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2142.350521] Lustre: server umount lustre-OST0000 complete [ 2143.712218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2143.717554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.719653] Lustre: Skipped 1 previous similar message [ 2148.320089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2148.322305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.327664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.787574] Lustre: server umount lustre-MDT0000 complete [ 2151.155190] LustreError: 120953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423081 with bad export cookie 12063719412725070612 [ 2151.159061] LustreError: 120953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2151.159122] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.292110] Lustre: server umount lustre-MDT0001 complete [ 2154.898178] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2155.881344] Key type lgssc unregistered [ 2156.006360] LNet: 123488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.008941] LNetError: 123488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2156.020280] LNet: Removed LNI 192.168.201.114@tcp [ 2156.293939] Key type .llcrypt unregistered [ 2156.295356] Key type ._llcrypt unregistered [ 2164.450799] Key type ._llcrypt registered [ 2164.451910] Key type .llcrypt registered [ 2164.492716] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2170.224278] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2170.633483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.655308] alg: No test for adler32 (adler32-zlib) [ 2171.500409] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 2171.579610] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2173.159158] Key type lgssc registered [ 2173.526827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2184.545459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2184.552116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.635471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2185.645478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2185.682229] Lustre: lustre-MDT0000: new disk, initializing [ 2185.706159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.713792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2187.030985] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2191.234288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.286132] Lustre: 126827:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2191.315106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2191.317178] Lustre: Skipped 1 previous similar message [ 2191.354728] Lustre: lustre-MDT0001: new disk, initializing [ 2191.390598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2191.405636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2191.410244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2192.677455] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2195.758026] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2197.036456] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2199.343676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2199.446537] Lustre: lustre-OST0000: new disk, initializing [ 2199.448855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2199.452270] Lustre: 128568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2199.483032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.353569] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2201.662220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2201.666732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2201.700451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2204.440595] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2205.797038] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2205.875044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2207.248895] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2207.327175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2210.075291] Lustre: server umount lustre-OST0000 complete [ 2211.808913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2211.809412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.812430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2211.816557] Lustre: Skipped 1 previous similar message [ 2216.928812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2216.932144] Lustre: Skipped 2 previous similar messages [ 2221.063720] Lustre: server umount lustre-MDT0000 complete [ 2222.279180] LustreError: 127604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423152 with bad export cookie 6914910552173436996 [ 2222.282138] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.283652] LustreError: 127604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2231.829312] Lustre: server umount lustre-MDT0001 complete [ 2234.667280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.815112] LustreError: 130637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.820975] LustreError: 130637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2234.834400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.124285] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2239.199447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.329861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2240.703347] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2242.477027] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2243.711070] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2246.156504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2248.085595] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2249.847534] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2251.135410] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2261.400214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2262.807966] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2262.900020] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2266.302079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2266.364556] Lustre: lustre-OST0001: new disk, initializing [ 2266.366397] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2266.368979] Lustre: 133705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2266.396231] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.399284] Lustre: Skipped 1 previous similar message [ 2268.291963] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2271.520427] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2275.862677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2275.867352] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2275.932198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2276.758242] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2279.666936] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2279.839728] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2282.670745] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d7a03eb3000.ost_server_uuid 50 [ 2283.742622] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d7a03eb3000.ost_server_uuid in FULL state after 0 sec [ 2291.168436] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2291.171130] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.176692] LustreError: Skipped 1 previous similar message [ 2291.180103] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2291.187327] Lustre: Skipped 1 previous similar message [ 2291.206666] Lustre: Skipped 1 previous similar message [ 2294.061695] Lustre: server umount lustre-OST0001 complete [ 2296.290678] LustreError: 132228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.318644] LustreError: 132228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2300.898745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2300.911173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.936279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2301.430651] LustreError: 132740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.450993] LustreError: 132740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2303.369619] Lustre: server umount lustre-OST0000 complete [ 2306.539412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.547047] Lustre: Skipped 1 previous similar message [ 2306.549708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2306.552150] Lustre: Skipped 2 previous similar messages [ 2311.136377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2311.146592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.548037] Lustre: server umount lustre-MDT0000 complete [ 2315.597665] LustreError: 130617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423245 with bad export cookie 6914910552173437864 [ 2315.599310] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.604318] LustreError: 130617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2315.848839] Lustre: server umount lustre-MDT0001 complete [ 2325.112968] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2326.457733] Key type lgssc unregistered [ 2326.616851] LNet: 136573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2326.626274] LNetError: 136573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.637707] LNet: Removed LNI 192.168.201.114@tcp [ 2327.097987] Key type .llcrypt unregistered [ 2327.099986] Key type ._llcrypt unregistered [ 2344.090542] Key type ._llcrypt registered [ 2344.091882] Key type .llcrypt registered [ 2344.141598] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2350.803178] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2351.303566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2351.319098] alg: No test for adler32 (adler32-zlib) [ 2352.232863] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 2352.329933] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2353.935406] Key type lgssc registered [ 2354.427834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2371.101754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2371.111254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.230815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2372.247835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2372.294307] Lustre: lustre-MDT0000: new disk, initializing [ 2372.343872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.356395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2374.180143] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2379.707166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.765902] Lustre: 140667:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2379.808037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2379.815392] Lustre: Skipped 1 previous similar message [ 2379.863594] Lustre: lustre-MDT0001: new disk, initializing [ 2379.905188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2379.923538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2379.927387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2381.557546] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2385.329966] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2386.826167] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2389.822663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2389.962699] Lustre: lustre-OST0000: new disk, initializing [ 2389.965495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2389.968833] Lustre: 142409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2390.005825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.951316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2391.955828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2392.014796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2392.598440] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2396.313528] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2398.380977] Lustre: Failing over lustre-OST0000 [ 2398.456037] Lustre: server umount lustre-OST0000 complete [ 2411.453639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2411.631702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2411.636959] Lustre: Skipped 1 previous similar message [ 2411.647966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.658440] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2411.703441] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2414.067057] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2415.947499] Lustre: Failing over lustre-MDT0000 [ 2416.056550] Lustre: server umount lustre-MDT0000 complete [ 2416.098759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2416.105518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.122136] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.128474] Lustre: Skipped 1 previous similar message [ 2418.143154] Lustre: 138891:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777423332/real 1777423332] req@00000000d5e66c2d x1863763395374080/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1777423348 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2421.452492] LustreError: 143014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.457406] LustreError: 143014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2422.240158] LustreError: 140678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.245812] LustreError: 140678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2423.266923] Lustre: 138892:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777423337/real 1777423337] req@000000007acc3e51 x1863763395375616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777423353 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2426.576131] LustreError: 143015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.958989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.014214] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.020072] Lustre: 140669:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.201.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2430.112120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2430.135759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.144353] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2430.144734] Lustre: lustre-OST0000: Recovery over after 0:19, of 3 clients 3 recovered and 0 were evicted. [ 2430.189412] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2431.695103] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2431.994260] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2435.558193] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2435.560091] Lustre: Skipped 1 previous similar message [ 2435.568335] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2435.586087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2440.673492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2440.673553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.680369] LustreError: Skipped 1 previous similar message [ 2440.686638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2440.689768] Lustre: Skipped 1 previous similar message [ 2442.847030] Lustre: server umount lustre-OST0000 complete [ 2445.796847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2445.796880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.802560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.811477] Lustre: Skipped 2 previous similar messages [ 2450.515103] Lustre: server umount lustre-MDT0000 complete [ 2450.917250] LustreError: 141727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.924762] LustreError: 141727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2451.999174] LustreError: 142408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423382 with bad export cookie 13371044697703268702 [ 2452.001552] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2452.003426] LustreError: 142408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2452.124579] Lustre: server umount lustre-MDT0001 complete [ 2456.135774] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2457.229462] Key type lgssc unregistered [ 2457.382117] LNet: 145896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.385448] LNetError: 145896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.394447] LNet: Removed LNI 192.168.201.114@tcp [ 2457.739688] Key type .llcrypt unregistered [ 2457.741279] Key type ._llcrypt unregistered [ 2478.801866] Key type ._llcrypt registered [ 2478.803334] Key type .llcrypt registered [ 2478.854498] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2479.289870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.322262] alg: No test for adler32 (adler32-zlib) [ 2480.251243] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 2480.390161] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2481.999196] Key type lgssc registered [ 2482.439904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.651216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.659261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.858995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.490413] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2492.058885] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2494.146752] LustreError: 147425:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2494.150207] LustreError: 147425:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2494.153666] LustreError: 147425:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2494.272975] Lustre: server umount lustre-MDT0000 complete [ 2499.767884] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2503.126469] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2515.787067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.879253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2515.890272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2515.927584] Lustre: lustre-MDT0000: new disk, initializing [ 2515.952240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.958948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2517.340762] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2521.724554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.770399] Lustre: 150575:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2521.798324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2521.800654] Lustre: Skipped 1 previous similar message [ 2521.841306] Lustre: lustre-MDT0001: new disk, initializing [ 2521.882716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2521.896034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2521.900499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2523.380949] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2526.732323] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2527.949210] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2530.459886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2530.571842] Lustre: lustre-OST0000: new disk, initializing [ 2530.574190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2530.578465] Lustre: 152318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2530.611286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.809756] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2536.251728] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2537.977183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2537.980581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2538.014963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2544.494445] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2545.043736] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:44:34 (1777423474) [ 2629.599528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2629.602569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.607847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.111672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.111848] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.231690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2635.234382] Lustre: Skipped 4 previous similar messages [ 2639.327124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2639.403306] Lustre: server umount lustre-MDT0000 complete [ 2640.351570] LustreError: 151359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.358131] LustreError: 151359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2640.804566] LustreError: 152317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423571 with bad export cookie 8924624019298197631 [ 2640.805635] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2640.810562] LustreError: 152317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2645.471400] LustreError: 150583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.471466] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.481225] Lustre: Skipped 1 previous similar message [ 2645.482991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2650.592663] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2650.592671] LustreError: 150582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.199100] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2655.271130] Lustre: server umount lustre-MDT0001 complete [ 2662.733994] Lustre: server umount lustre-OST0000 complete [ 2665.511616] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2668.276587] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2679.697630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.780523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2679.789890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2679.833615] Lustre: lustre-MDT0000: new disk, initializing [ 2679.857022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.863725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2681.133515] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2685.181611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.218142] Lustre: 156801:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2685.238542] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2685.240907] Lustre: Skipped 1 previous similar message [ 2685.274777] Lustre: lustre-MDT0001: new disk, initializing [ 2685.317364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2685.320441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2686.511609] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2689.435634] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2690.522962] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2692.536576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2692.629586] Lustre: 158544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2692.652206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.654987] Lustre: Skipped 1 previous similar message [ 2694.356599] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2697.445185] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2698.761319] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2700.790742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2700.793942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2700.824178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2700.881810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2702.253542] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2702.343164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2705.893183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.893230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2705.894414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2705.894419] Lustre: Skipped 1 previous similar message [ 2705.897167] Lustre: Skipped 1 previous similar message [ 2705.908854] LustreError: Skipped 1 previous similar message [ 2709.074916] Lustre: server umount lustre-OST0000 complete [ 2711.008842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.008945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2711.013186] Lustre: Skipped 1 previous similar message [ 2716.507121] Lustre: server umount lustre-MDT0000 complete [ 2718.005678] LustreError: 156791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423648 with bad export cookie 8924624019298356447 [ 2718.009783] LustreError: 156791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2718.010660] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2718.146203] Lustre: server umount lustre-MDT0001 complete [ 2721.064408] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:47:30 (1777423650) [ 2726.275511] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2729.469944] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2741.121659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.212561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2741.223441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2741.225616] Lustre: Skipped 1 previous similar message [ 2741.261133] Lustre: lustre-MDT0000: new disk, initializing [ 2741.262661] Lustre: Skipped 1 previous similar message [ 2741.285640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.292601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2742.617235] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2746.643659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.684602] Lustre: 163122:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2746.780552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2748.021670] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2751.022278] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2752.164912] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2754.386658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2754.523794] Lustre: 164872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2756.390028] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2759.402343] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2760.692381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2760.722057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2770.914303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.914475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2770.920064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2770.922063] LustreError: Skipped 1 previous similar message [ 2770.923906] Lustre: Skipped 5 previous similar messages [ 2772.686529] Lustre: server umount lustre-OST0000 complete [ 2788.319118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2788.383130] Lustre: server umount lustre-MDT0000 complete [ 2789.790231] LustreError: 163904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423720 with bad export cookie 8924624019298357329 [ 2789.792945] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2789.794329] LustreError: 163904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2789.926638] Lustre: server umount lustre-MDT0001 complete [ 2793.157679] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:48:42 (1777423722) [ 2799.177085] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2802.490128] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2815.761574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.875268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2815.890599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2815.893655] Lustre: Skipped 3 previous similar messages [ 2815.932117] Lustre: lustre-MDT0000: new disk, initializing [ 2815.934175] Lustre: Skipped 2 previous similar messages [ 2815.968383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.971196] Lustre: Skipped 2 previous similar messages [ 2815.978932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2815.982378] Lustre: Skipped 2 previous similar messages [ 2817.544152] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2822.287383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.365629] Lustre: 169043:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2822.505957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2823.939770] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2827.282892] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2828.595710] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2831.245858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2831.381358] Lustre: 170791:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2833.343752] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2836.924572] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2841.641512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2846.689531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2846.690336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.693163] LustreError: Skipped 1 previous similar message [ 2846.693850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2846.700955] Lustre: Skipped 4 previous similar messages [ 2846.712333] Lustre: Skipped 7 previous similar messages [ 2850.447196] Lustre: server umount lustre-OST0000 complete [ 2866.144368] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2866.230421] Lustre: server umount lustre-MDT0000 complete [ 2867.170419] LustreError: 169053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.178128] LustreError: 169053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2867.722400] LustreError: 169034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423798 with bad export cookie 8924624019298358400 [ 2867.726342] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2867.726582] LustreError: 169034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2867.855863] Lustre: server umount lustre-MDT0001 complete [ 2871.804177] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2872.978618] Key type lgssc unregistered [ 2873.107357] LNet: 172516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2873.109779] LNetError: 172516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2873.119307] LNet: Removed LNI 192.168.201.114@tcp [ 2873.453922] Key type .llcrypt unregistered [ 2873.455485] Key type ._llcrypt unregistered [ 2883.356599] Key type ._llcrypt registered [ 2883.357992] Key type .llcrypt registered [ 2883.403337] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2883.762963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2883.798065] alg: No test for adler32 (adler32-zlib) [ 2884.661053] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 2884.748450] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2886.335207] Key type lgssc registered [ 2886.705119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2889.488863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2889.500390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.677488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.277461] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2895.535807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.678632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2897.123126] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2899.129107] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2900.433657] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2903.116234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2903.251907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.260607] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2907.055576] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2916.583125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2932.194534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.194735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2932.199850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2932.203555] Lustre: Skipped 1 previous similar message [ 2935.324637] Lustre: server umount lustre-OST0000 complete [ 2936.801080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2936.804011] LustreError: Skipped 1 previous similar message [ 2936.805448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.810022] Lustre: Skipped 1 previous similar message [ 2936.812198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.432338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.434201] Lustre: Skipped 3 previous similar messages [ 2947.551878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.553959] Lustre: Skipped 1 previous similar message [ 2951.135137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2951.208282] Lustre: server umount lustre-MDT0000 complete [ 2951.649772] LustreError: 173894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.656254] LustreError: 173894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2952.585275] LustreError: 176061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423882 with bad export cookie 15619986474957811438 [ 2952.588896] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2952.589421] LustreError: 176061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2952.710697] Lustre: server umount lustre-MDT0001 complete [ 2956.438441] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2957.509893] Key type lgssc unregistered [ 2957.655551] LNet: 177097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2957.658445] LNetError: 177097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2957.669325] LNet: Removed LNI 192.168.201.114@tcp [ 2957.995124] Key type .llcrypt unregistered [ 2957.996651] Key type ._llcrypt unregistered [ 2967.043976] Key type ._llcrypt registered [ 2967.045319] Key type .llcrypt registered [ 2967.087416] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2967.473517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2967.502175] alg: No test for adler32 (adler32-zlib) [ 2968.367070] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 2968.471591] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2970.063234] Key type lgssc registered [ 2970.450307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2973.213593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2973.219695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.360389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.664861] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2978.657855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.782508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2980.065841] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2981.982862] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2983.310743] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2985.982826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.116655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2988.181657] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2990.091440] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2999.461491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3004.896836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3004.898570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.900622] LustreError: Skipped 1 previous similar message [ 3004.906239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3004.909777] Lustre: Skipped 1 previous similar message [ 3005.654401] Lustre: lustre-OST0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 3010.016818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.019239] Lustre: Skipped 1 previous similar message [ 3015.139508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3015.141668] Lustre: Skipped 2 previous similar messages [ 3017.183148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3017.236647] Lustre: server umount lustre-OST0000 complete [ 3019.125134] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9d7a0b2f6000.ost_server_uuid 50 [ 3019.690899] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d7a0b2f6000.ost_server_uuid in DISCONN state after 0 sec [ 3020.253357] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3030.252569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3030.253134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3030.256519] Lustre: Skipped 2 previous similar messages [ 3030.258713] Lustre: Skipped 1 previous similar message [ 3036.370199] Lustre: server umount lustre-MDT0000 complete [ 3037.681199] LustreError: 178455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777423968 with bad export cookie 6004486369224378046 [ 3037.684168] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3037.686241] LustreError: 178455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3037.801262] Lustre: server umount lustre-MDT0001 complete [ 3040.642634] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:52:50 (1777423970) [ 3042.818802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.955473] LustreError: 182003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.961292] LustreError: 182003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3042.978335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.196469] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3047.141506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3047.259327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3048.585193] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3050.411915] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3051.631220] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3053.931761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3055.954437] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3057.818982] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3060.014701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3060.085043] Lustre: lustre-OST0001: new disk, initializing [ 3060.087916] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3060.091780] Lustre: 184501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3060.123248] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3060.126102] Lustre: Skipped 1 previous similar message [ 3060.480422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3060.482917] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3060.525679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3061.944919] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3065.070310] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3067.359801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3067.362611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.367658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3067.370020] Lustre: Skipped 3 previous similar messages [ 3071.998295] Lustre: server umount lustre-OST0000 complete [ 3073.505744] LustreError: 183596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.511947] LustreError: 183596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3073.980543] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3074.064844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3075.622068] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3075.714495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3075.790127] LustreError: 183597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.625043] LustreError: 185201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.629134] LustreError: 185201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3083.748658] LustreError: 183595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.754077] LustreError: 183595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3086.304105] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3086.304351] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.306020] LustreError: Skipped 1 previous similar message [ 3086.310207] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3086.312995] Lustre: Skipped 4 previous similar messages [ 3091.860980] Lustre: server umount lustre-OST0001 complete [ 3093.471741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3093.474221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.477661] Lustre: Skipped 1 previous similar message [ 3099.284191] Lustre: server umount lustre-MDT0000 complete [ 3100.447053] LustreError: 182028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424030 with bad export cookie 6004486369224379103 [ 3100.450089] LustreError: 182028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3100.450138] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3100.559381] Lustre: server umount lustre-MDT0001 complete [ 3108.101968] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:53:57 (1777424037) [ 3110.388803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.477648] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3110.488760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3110.558846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.776138] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3114.647248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.684463] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3114.696848] Lustre: 187775:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3116.080162] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3117.782885] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3118.969360] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3121.142655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3121.214061] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3123.122943] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3124.784567] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3127.050074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3127.073918] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3128.361926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3128.373699] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3128.972830] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3130.840962] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3133.411309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.411924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.414126] LustreError: Skipped 1 previous similar message [ 3133.423680] Lustre: Skipped 1 previous similar message [ 3133.425901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.431252] Lustre: Skipped 7 previous similar messages [ 3137.980397] Lustre: server umount lustre-OST0000 complete [ 3138.531737] LustreError: 190502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.536277] LustreError: 190502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3148.768327] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3154.771843] Lustre: server umount lustre-OST0001 complete [ 3159.008868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.012351] Lustre: Skipped 3 previous similar messages [ 3162.198809] Lustre: server umount lustre-MDT0000 complete [ 3163.397164] LustreError: 187765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424093 with bad export cookie 6004486369224380237 [ 3163.399945] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.401021] LustreError: 187765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3163.514175] Lustre: server umount lustre-MDT0001 complete [ 3171.124964] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:55:00 (1777424100) [ 3176.692777] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 3179.768170] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3191.819380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.904992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3191.916562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3191.952742] Lustre: lustre-MDT0000: new disk, initializing [ 3191.974869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.977549] Lustre: Skipped 3 previous similar messages [ 3191.983454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3193.285675] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3197.445663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.485481] Lustre: 195344:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3197.508706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3197.510874] Lustre: Skipped 1 previous similar message [ 3197.547889] Lustre: lustre-MDT0001: new disk, initializing [ 3197.595592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3197.599091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3198.900380] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3201.995325] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3203.245591] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3205.507174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.614372] Lustre: lustre-OST0000: new disk, initializing [ 3205.616775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3205.619155] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3205.621053] Lustre: Skipped 7 previous similar messages [ 3205.621222] Lustre: 197087:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3207.610360] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3210.687762] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3210.734567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3210.738880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3210.754128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3212.046474] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3212.121833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3213.384490] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3213.460136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3215.841707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.841776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.844568] LustreError: Skipped 2 previous similar messages [ 3220.243737] Lustre: server umount lustre-OST0000 complete [ 3227.676764] Lustre: server umount lustre-MDT0000 complete [ 3228.868151] LustreError: 195335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424159 with bad export cookie 6004486369224381903 [ 3228.871467] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3228.872163] LustreError: 195335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3228.984509] Lustre: server umount lustre-MDT0001 complete [ 3231.960154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.113345] LustreError: 199161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.121083] LustreError: 199161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3233.429690] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3236.389303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.894779] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3239.849737] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.220205] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3243.359884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3245.361856] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3247.161260] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3248.558963] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3258.812372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3260.057729] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3260.126878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3261.919830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3261.922204] LustreError: Skipped 1 previous similar message [ 3266.824362] Lustre: server umount lustre-OST0000 complete [ 3268.581628] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3269.780800] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3293.256411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.359586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.362760] Lustre: Skipped 5 previous similar messages [ 3295.194892] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3296.834480] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3297.328030] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3304.341021] Lustre: server umount lustre-OST0000 complete [ 3308.512578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.516856] Lustre: Skipped 6 previous similar messages [ 3311.697134] Lustre: server umount lustre-MDT0000 complete [ 3312.908158] LustreError: 199141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424243 with bad export cookie 6004486369224382785 [ 3312.911099] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3312.911773] LustreError: 199141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3313.015940] Lustre: server umount lustre-MDT0001 complete [ 3315.853343] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:57:25 (1777424245) [ 3318.214067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.369900] LustreError: 204672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.709132] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3322.463194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.859400] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3325.729814] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3327.044642] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3329.377466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3331.381374] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3333.210727] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3334.665912] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3342.881343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3344.098250] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3344.168318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3346.317023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3346.383963] Lustre: lustre-OST0001: new disk, initializing [ 3346.385931] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3346.388653] Lustre: 207725:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3348.185451] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3351.332833] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3352.689770] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3353.595488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3353.599431] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3353.623611] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3353.779632] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3355.091486] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3355.162477] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3358.690415] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3358.692561] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3358.692875] LustreError: Skipped 3 previous similar messages [ 3358.694766] Lustre: Skipped 11 previous similar messages [ 3361.918736] Lustre: server umount lustre-OST0001 complete [ 3363.756709] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3363.825721] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3365.017980] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3365.086545] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3384.290141] LustreError: 206266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.294934] LustreError: 206266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3388.756402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.683031] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3392.480650] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3393.088265] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3399.797887] Lustre: server umount lustre-OST0001 complete [ 3426.271112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3426.322846] Lustre: server umount lustre-OST0000 complete [ 3433.879831] Lustre: server umount lustre-MDT0000 complete [ 3435.152086] LustreError: 204698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424365 with bad export cookie 6004486369224383716 [ 3435.155934] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.156591] LustreError: 204698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3441.337182] Lustre: server umount lustre-MDT0001 complete [ 3449.085531] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:59:38 (1777424378) [ 3451.385057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.482895] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3451.494359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3451.568512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3451.571897] Lustre: Skipped 5 previous similar messages [ 3452.913064] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3455.790655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.824462] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3455.833439] Lustre: 212678:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3457.195370] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3459.039615] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3460.344417] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3462.742440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3462.808180] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3464.905029] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3466.853317] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3470.809362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.837678] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3472.782639] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3474.508052] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3475.039330] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3476.354099] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3476.424802] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3477.691511] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3477.766650] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3478.993109] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d7a03ecf800.ost_server_uuid 50 [ 3479.492285] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d7a03ecf800.ost_server_uuid in FULL state after 0 sec [ 3479.831108] Lustre: Permanently deactivating lustre-OST0001 [ 3479.836336] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3480.644970] Lustre: Permanently reactivating lustre-OST0001 [ 3486.179641] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.184264] Lustre: Skipped 7 previous similar messages [ 3487.737815] Lustre: server umount lustre-OST0001 complete [ 3495.062715] Lustre: server umount lustre-OST0000 complete [ 3502.498237] Lustre: server umount lustre-MDT0000 complete [ 3503.761124] LustreError: 212668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424434 with bad export cookie 6004486369224384976 [ 3503.764486] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3503.765077] LustreError: 212668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3503.879581] Lustre: server umount lustre-MDT0001 complete [ 3511.717381] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:00:41 (1777424441) [ 3514.208526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.317947] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3514.328270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3514.331405] Lustre: Skipped 5 previous similar messages [ 3515.717794] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3518.585898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3518.619726] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3518.629689] Lustre: 218775:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3519.959993] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3521.719555] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.939216] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3525.153881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3525.226422] Lustre: Permanently deactivating lustre-OST0000 [ 3525.230803] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3527.135258] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3528.914537] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3529.135178] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3531.297500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3533.231719] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3534.996735] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3535.128240] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3535.554152] Lustre: Permanently reactivating lustre-OST0000 [ 3543.540490] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3543.565217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3550.689621] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3550.693457] LustreError: Skipped 5 previous similar messages [ 3554.051942] Lustre: server umount lustre-OST0001 complete [ 3555.809404] LustreError: 222055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3555.814748] LustreError: 222055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 3561.486775] Lustre: server umount lustre-OST0000 complete [ 3577.311122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3577.374042] Lustre: server umount lustre-MDT0000 complete [ 3578.682226] LustreError: 218765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424509 with bad export cookie 6004486369224386607 [ 3578.685437] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3578.687263] LustreError: 218765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3578.810228] Lustre: server umount lustre-MDT0001 complete [ 3582.597440] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3583.645451] Key type lgssc unregistered [ 3583.776351] LNet: 223193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3583.779027] LNetError: 223193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3583.787291] LNet: Removed LNI 192.168.201.114@tcp [ 3584.088413] Key type .llcrypt unregistered [ 3584.089838] Key type ._llcrypt unregistered [ 3592.573704] Key type ._llcrypt registered [ 3592.574794] Key type .llcrypt registered [ 3592.610450] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3592.989801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3593.007980] alg: No test for adler32 (adler32-zlib) [ 3593.852710] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 3593.933798] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3595.511157] Key type lgssc registered [ 3595.877457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3599.105276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3599.110862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.264612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.518312] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3604.264718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.303072] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3604.313929] Lustre: 224609:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3604.318520] Lustre: Permanently deactivating lustre-MDT0001 [ 3604.323915] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3604.431878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3605.808122] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3607.340613] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3607.342982] LustreError: 224652:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3607.346629] LustreError: 224652:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3607.350545] LustreError: 224652:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3607.570459] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3608.800055] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3611.013744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3611.127117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.954611] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3614.179097] LustreError: 226236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.182218] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3614.184706] LustreError: 226236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3614.695319] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3617.034363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3617.111288] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3618.958090] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3620.779247] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3622.373550] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3625.447134] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3625.447484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3626.619223] Lustre: Permanently reactivating lustre-MDT0001 [ 3626.624124] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3626.626394] Lustre: Skipped 2 previous similar messages [ 3630.358555] Lustre: lustre-MDT0001: Client 80aa0f22-b3f5-4ceb-9a13-31f2d9f0ae61 (at 192.168.201.14@tcp) reconnecting [ 3632.025276] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3632.589359] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3634.292311] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3634.371267] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3635.696468] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3635.775162] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3636.604964] Lustre: Permanently deactivating lustre-MDT0001 [ 3636.611334] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3636.615101] Lustre: Skipped 2 previous similar messages [ 3641.855326] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3645.407933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3645.411395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.417382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3645.920916] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.922836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3645.925079] Lustre: Skipped 2 previous similar messages [ 3645.928659] Lustre: Skipped 2 previous similar messages [ 3650.523405] Lustre: server umount lustre-MDT0000 complete [ 3651.041307] LustreError: 224621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.046696] LustreError: 224621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3651.765978] LustreError: 224647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424582 with bad export cookie 8300900116662235640 [ 3651.769287] LustreError: 224647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3651.769380] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.876313] Lustre: server umount lustre-MDT0001 complete [ 3659.195925] Lustre: server umount lustre-OST0000 complete [ 3663.000689] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:03:12 (1777424592) [ 3666.599315] Lustre: server umount lustre-OST0001 complete [ 3668.734338] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 3671.780625] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3684.005585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.100032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3684.103667] Lustre: Skipped 2 previous similar messages [ 3684.113867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3684.152736] Lustre: lustre-MDT0000: new disk, initializing [ 3684.178141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.185180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3685.622494] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3689.844860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.891327] Lustre: 232304:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3689.914991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3689.917233] Lustre: Skipped 1 previous similar message [ 3689.952141] Lustre: lustre-MDT0001: new disk, initializing [ 3689.993317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3689.997995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3691.362389] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3694.624685] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3695.991699] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3698.550809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3698.658341] Lustre: lustre-OST0000: new disk, initializing [ 3698.660718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3698.664854] Lustre: 234046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3698.692236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3698.695584] Lustre: Skipped 1 previous similar message [ 3700.691708] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3702.582835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3702.586140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3702.622599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3703.944468] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3704.607652] LustreError: 232309:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3708.792502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.855558] Lustre: lustre-OST0001: new disk, initializing [ 3708.858418] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3708.862096] Lustre: 235047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3710.860043] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3711.041065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3711.046766] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3711.078369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3714.190396] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3714.695155] LustreError: 232309:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3714.702973] LustreError: 232309:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3714.777360] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3724.791097] LustreError: 232309:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3726.307105] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.307702] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3726.311209] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3726.314746] Lustre: Skipped 1 previous similar message [ 3730.644290] Lustre: lustre-OST0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 3731.774063] Lustre: server umount lustre-OST0001 complete [ 3736.546046] LustreError: 234031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.550634] LustreError: 234031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3741.666927] LustreError: 234527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.670891] LustreError: 234527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3746.785752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.785884] LustreError: 234527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3746.787595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3746.787602] Lustre: Skipped 2 previous similar messages [ 3746.789381] Lustre: Skipped 2 previous similar messages [ 3746.794574] LustreError: 234527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3757.025931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.026738] LustreError: 234030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3757.028018] Lustre: Skipped 4 previous similar messages [ 3757.036071] LustreError: 234030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3758.047076] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3758.094966] Lustre: server umount lustre-OST0000 complete [ 3761.631923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3761.635278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.590597] Lustre: server umount lustre-MDT0000 complete [ 3766.818090] LustreError: 233082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424697 with bad export cookie 8300900116662238454 [ 3766.820915] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3766.822608] LustreError: 233082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3781.087113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3781.177713] Lustre: server umount lustre-MDT0001 complete [ 3784.824191] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3785.946908] Key type lgssc unregistered [ 3786.091491] LNet: 236965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3786.093986] LNetError: 236965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3786.102503] LNet: Removed LNI 192.168.201.114@tcp [ 3786.425314] Key type .llcrypt unregistered [ 3786.426387] Key type ._llcrypt unregistered [ 3799.990491] Key type ._llcrypt registered [ 3799.991566] Key type .llcrypt registered [ 3800.026686] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3800.440819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3800.450549] alg: No test for adler32 (adler32-zlib) [ 3801.312630] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 3801.397163] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3802.975129] Key type lgssc registered [ 3803.302419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3805.721163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3805.726658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.816981] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3806.824262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3806.877971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3808.208409] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3811.055577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.086602] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3811.097274] Lustre: 239133:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3811.184393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3812.420701] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3814.167287] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3815.390887] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3817.576799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3817.640795] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3817.704094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.578269] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3821.430396] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3823.127680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3869.153123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3869.153482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3869.154490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3869.155232] LustreError: Skipped 1 previous similar message [ 3869.161112] Lustre: Skipped 1 previous similar message [ 3871.642089] Lustre: server umount lustre-OST0000 complete [ 3873.015041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3877.053655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3877.172300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.111300] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3881.003143] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3889.443517] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3889.446498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3889.446531] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3889.451059] Lustre: Skipped 1 previous similar message [ 3915.232559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3915.233979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.234722] LustreError: Skipped 1 previous similar message [ 3915.235226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3915.239051] Lustre: Skipped 1 previous similar message [ 3915.240417] Lustre: Skipped 1 previous similar message [ 3920.282481] Lustre: server umount lustre-OST0000 complete [ 3923.935759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3923.938183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.942509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3923.944273] Lustre: Skipped 1 previous similar message [ 3927.712584] Lustre: server umount lustre-MDT0000 complete [ 3928.932072] LustreError: 239125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424859 with bad export cookie 16749779330491202900 [ 3928.935718] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3928.936161] LustreError: 239125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3929.063958] Lustre: server umount lustre-MDT0001 complete [ 3932.712627] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3933.780941] Key type lgssc unregistered [ 3933.907477] LNet: 244080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3933.910203] LNetError: 244080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3933.921371] LNet: Removed LNI 192.168.201.114@tcp [ 3934.211858] Key type .llcrypt unregistered [ 3934.213077] Key type ._llcrypt unregistered [ 3943.403715] Key type ._llcrypt registered [ 3943.404846] Key type .llcrypt registered [ 3943.450857] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3943.865524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3943.897067] alg: No test for adler32 (adler32-zlib) [ 3944.779752] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 3944.891303] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3946.487246] Key type lgssc registered [ 3946.887459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3949.616344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3949.622640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.772186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.978202] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3954.720744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3954.830192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.025894] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3957.717141] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3958.920092] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3961.049770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3961.156352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.214129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3962.924669] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3964.643287] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3975.650154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3975.650940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3975.652577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3975.655090] Lustre: Skipped 1 previous similar message [ 3978.696911] Lustre: server umount lustre-OST0000 complete [ 3980.768678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.769074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3980.769785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3980.769790] Lustre: Skipped 1 previous similar message [ 3980.772719] Lustre: Skipped 1 previous similar message [ 3985.887945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3985.889472] Lustre: Skipped 2 previous similar messages [ 3986.002663] Lustre: server umount lustre-MDT0000 complete [ 3987.155407] LustreError: 246175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424917 with bad export cookie 17902295032815033484 [ 3987.159425] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.159729] LustreError: 246175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3987.267849] Lustre: server umount lustre-MDT0001 complete [ 3990.679478] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3991.703583] Key type lgssc unregistered [ 3991.824436] LNet: 249404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3991.827519] LNetError: 249404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3991.836311] LNet: Removed LNI 192.168.201.114@tcp [ 3992.126892] Key type .llcrypt unregistered [ 3992.127933] Key type ._llcrypt unregistered [ 3999.033048] Key type ._llcrypt registered [ 3999.034098] Key type .llcrypt registered [ 3999.072159] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3999.403866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3999.429898] alg: No test for adler32 (adler32-zlib) [ 4000.291945] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 4000.371899] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4001.951101] Key type lgssc registered [ 4002.309064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4005.006870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4005.012557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.179653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.557952] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4010.565292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4010.686127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4011.943630] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4013.768814] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4014.960131] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4017.186295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4017.298917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.089377] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4020.900731] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4031.653352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4042.208995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.210456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.212320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.212893] Lustre: Skipped 1 previous similar message [ 4046.039752] Lustre: server umount lustre-OST0000 complete [ 4047.328255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.328904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4047.332090] Lustre: Skipped 1 previous similar message [ 4047.334616] Lustre: Skipped 1 previous similar message [ 4052.448040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4052.450840] Lustre: Skipped 2 previous similar messages [ 4053.397285] Lustre: server umount lustre-MDT0000 complete [ 4054.560176] LustreError: 250568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777424984 with bad export cookie 11214121148459004566 [ 4054.562424] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4054.565654] LustreError: 250568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4054.663937] Lustre: server umount lustre-MDT0001 complete [ 4058.122568] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4059.139378] Key type lgssc unregistered [ 4059.268378] LNet: 253929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4059.271643] LNetError: 253929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4059.280293] LNet: Removed LNI 192.168.201.114@tcp [ 4059.575883] Key type .llcrypt unregistered [ 4059.577587] Key type ._llcrypt unregistered [ 4068.219874] Key type ._llcrypt registered [ 4068.221243] Key type .llcrypt registered [ 4068.259901] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4068.614335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4068.651660] alg: No test for adler32 (adler32-zlib) [ 4069.505939] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 4069.593425] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4071.167141] Key type lgssc registered [ 4071.502260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4073.900802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4073.906611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.049978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.264608] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4079.013051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.143421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4080.353176] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4081.970403] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4083.092600] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4085.348932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.459560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.321747] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4089.079988] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4099.813302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4112.865033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4112.867996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.873404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.426190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.426843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.428779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4118.160384] Lustre: server umount lustre-OST0000 complete [ 4120.031767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4120.034583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.039371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4120.041564] Lustre: Skipped 1 previous similar message [ 4125.523096] Lustre: server umount lustre-MDT0000 complete [ 4125.663976] LustreError: 255307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.669118] LustreError: 255307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4126.830412] LustreError: 255286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425057 with bad export cookie 4722894558631052991 [ 4126.834733] LustreError: 255286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4126.834798] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4126.960201] Lustre: server umount lustre-MDT0001 complete [ 4130.510156] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4131.531504] Key type lgssc unregistered [ 4131.658529] LNet: 259276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4131.661943] LNetError: 259276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4131.671292] LNet: Removed LNI 192.168.201.114@tcp [ 4131.986376] Key type .llcrypt unregistered [ 4131.988246] Key type ._llcrypt unregistered [ 4138.776405] Key type ._llcrypt registered [ 4138.778858] Key type .llcrypt registered [ 4138.820205] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4139.242372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4139.252974] alg: No test for adler32 (adler32-zlib) [ 4140.110654] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 4140.196419] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4141.783189] Key type lgssc registered [ 4142.165956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4144.854749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4144.863321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.045548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.329687] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4150.103322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.222046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4151.465858] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4153.199924] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4154.377531] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4156.576209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.686536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.665517] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4160.470782] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4171.046318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4180.447883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4180.450495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.455523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4181.473332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4181.473503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4181.474247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.484231] Lustre: Skipped 1 previous similar message [ 4185.806541] Lustre: server umount lustre-OST0000 complete [ 4191.199798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4191.202261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.206345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4193.241331] Lustre: server umount lustre-MDT0000 complete [ 4194.524496] LustreError: 260439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425124 with bad export cookie 3948801620653636630 [ 4194.528519] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4194.528742] LustreError: 260439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4194.674576] Lustre: server umount lustre-MDT0001 complete [ 4198.196456] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4199.191682] Key type lgssc unregistered [ 4199.312522] LNet: 263840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4199.315661] LNetError: 263840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4199.322307] LNet: Removed LNI 192.168.201.114@tcp [ 4199.617523] Key type .llcrypt unregistered [ 4199.619579] Key type ._llcrypt unregistered [ 4203.355704] Key type ._llcrypt registered [ 4203.356820] Key type .llcrypt registered [ 4203.395858] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4209.029253] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 4214.943933] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4215.276362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4215.305582] alg: No test for adler32 (adler32-zlib) [ 4216.172314] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 4216.258344] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4217.839142] Key type lgssc registered [ 4218.202212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4229.295758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4229.302916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4230.398753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4230.410789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4230.446882] Lustre: lustre-MDT0000: new disk, initializing [ 4230.476884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.483475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4231.749267] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4235.727148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4235.769913] Lustre: 267314:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4235.795663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4235.797422] Lustre: Skipped 1 previous similar message [ 4235.830705] Lustre: lustre-MDT0001: new disk, initializing [ 4235.860500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4235.872614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4235.876413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4237.228233] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4240.211710] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4241.281767] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4243.441403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4243.533699] Lustre: lustre-OST0000: new disk, initializing [ 4243.535924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4243.538646] Lustre: 269055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4243.563168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.355359] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4248.299234] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4248.566838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4248.569982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4248.600950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4249.536434] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4249.602695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4250.779524] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4250.846084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4253.666177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4253.666314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.667800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4253.668528] LustreError: Skipped 1 previous similar message [ 4253.672792] Lustre: Skipped 1 previous similar message [ 4257.554197] Lustre: server umount lustre-OST0000 complete [ 4261.343911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4261.346055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.350464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4261.352497] Lustre: Skipped 1 previous similar message [ 4263.904168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4263.904451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.024045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.026377] Lustre: Skipped 2 previous similar messages [ 4273.119114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4273.177640] Lustre: server umount lustre-MDT0000 complete [ 4274.428196] LustreError: 269054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425204 with bad export cookie 1019573975768944038 [ 4274.431434] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4274.431970] LustreError: 269054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4274.542700] Lustre: server umount lustre-MDT0001 complete [ 4277.093371] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:13:26 (1777425206) [ 4279.263211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4279.417788] LustreError: 271319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.424389] LustreError: 271319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4279.443083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.745931] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4283.462706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.578171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4284.732715] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4286.302023] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4287.425197] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4289.466033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4291.260639] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4292.942928] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4314.081442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4314.082685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.085076] LustreError: Skipped 1 previous similar message [ 4314.091149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4314.094895] Lustre: Skipped 1 previous similar message [ 4317.845385] Lustre: server umount lustre-OST0000 complete [ 4319.200327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.205075] Lustre: Skipped 2 previous similar messages [ 4324.320074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4324.322248] Lustre: Skipped 3 previous similar messages [ 4325.341466] Lustre: server umount lustre-MDT0000 complete [ 4326.589192] LustreError: 271300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425256 with bad export cookie 1019573975768944906 [ 4326.593534] LustreError: 271300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4326.593743] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.596119] LustreError: 272039:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4326.728465] Lustre: server umount lustre-MDT0001 complete [ 4330.450757] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4331.513148] Key type lgssc unregistered [ 4331.651415] LNet: 274518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4331.654057] LNetError: 274518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4331.663345] LNet: Removed LNI 192.168.201.114@tcp [ 4331.986885] Key type .llcrypt unregistered [ 4331.988426] Key type ._llcrypt unregistered [ 4339.972735] Key type ._llcrypt registered [ 4339.973788] Key type .llcrypt registered [ 4340.009277] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 4345.671909] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4346.060030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4346.088650] alg: No test for adler32 (adler32-zlib) [ 4346.927825] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 4347.006110] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4348.583148] Key type lgssc registered [ 4348.915860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4362.255953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4362.261961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.344387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4363.353897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4363.386537] Lustre: lustre-MDT0000: new disk, initializing [ 4363.408387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.414779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4364.599035] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4368.644178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.689267] Lustre: 278281:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4368.715658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4368.717498] Lustre: Skipped 1 previous similar message [ 4368.756656] Lustre: lustre-MDT0001: new disk, initializing [ 4368.793936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4368.807968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4368.811429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4370.057063] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4373.111343] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4374.399539] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4376.683691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4376.789942] Lustre: lustre-OST2710: new disk, initializing [ 4376.791777] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4376.794125] Lustre: 280022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4376.825241] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4378.700632] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4381.779542] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4382.710598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4382.713659] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4382.748472] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4384.143138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4384.212831] Lustre: lustre-OST03e8: new disk, initializing [ 4384.214813] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4384.217647] Lustre: 280971:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4384.251865] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4386.080168] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4387.843638] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4390.399086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4390.403180] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4390.438778] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4390.450173] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4390.525265] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4391.825362] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4391.906327] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4393.084308] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4393.148972] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4394.319708] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4394.382569] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4395.491488] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4395.493947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4395.496358] Lustre: Skipped 3 previous similar messages [ 4395.498536] Lustre: Skipped 2 previous similar messages [ 4400.608930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.611393] Lustre: Skipped 2 previous similar messages [ 4401.437232] Lustre: server umount lustre-MDT0000 complete [ 4402.690339] LustreError: 278272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425333 with bad export cookie 13017366279113076584 [ 4402.693851] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4402.694224] LustreError: 278272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4402.813474] Lustre: server umount lustre-MDT0001 complete [ 4413.955986] Lustre: server umount lustre-OST2710 complete [ 4425.442795] Lustre: server umount lustre-OST03e8 complete [ 4431.311145] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 4434.502764] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4446.711598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.807337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4446.819739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4446.858117] Lustre: lustre-MDT0000: new disk, initializing [ 4446.883329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.891814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4448.345077] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4452.595799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.642625] Lustre: 285998:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4452.745849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4454.086446] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4457.218337] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4458.472718] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4460.747358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4460.860124] Lustre: lustre-OST0000: new disk, initializing [ 4460.861564] Lustre: Skipped 1 previous similar message [ 4460.863885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4460.866591] Lustre: Skipped 2 previous similar messages [ 4460.869039] Lustre: 287738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4460.896641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4460.899936] Lustre: Skipped 1 previous similar message [ 4461.217352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4461.221484] Lustre: Skipped 1 previous similar message [ 4461.223866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4461.260918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4462.859995] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4465.979941] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4467.388404] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4467.463399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4468.836795] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4468.918153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4471.777809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4471.778059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.781084] LustreError: Skipped 1 previous similar message [ 4471.787689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4471.792396] Lustre: Skipped 1 previous similar message [ 4475.664899] Lustre: server umount lustre-OST0000 complete [ 4478.431895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4478.434240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.437100] Lustre: Skipped 1 previous similar message [ 4478.438548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.017726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.163864] Lustre: server umount lustre-MDT0000 complete [ 4484.315230] LustreError: 285989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425414 with bad export cookie 13017366279113078243 [ 4484.318466] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4484.319328] LustreError: 285989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4484.421470] Lustre: server umount lustre-MDT0001 complete [ 4486.908905] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:16:56 (1777425416) [ 4487.349808] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4487.838752] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:16:57 (1777425417) [ 4488.732431] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4497.072963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.163189] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4497.173475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4497.233210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.435378] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4501.188447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4501.221260] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4501.230840] Lustre: 291424:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4502.556140] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4504.239961] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4505.344444] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4518.484264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4518.564909] Lustre: Denying initial registration attempt from nid 192.168.201.114@tcp, specified as failover [ 4518.568781] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4518.571486] LustreError: 293037:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 4518.573636] LustreError: 293037:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4518.575506] LustreError: 293037:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4518.600397] Lustre: server umount lustre-OST0000 complete [ 4518.602624] LustreError: 293037:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4521.440556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.441288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4521.444855] Lustre: Skipped 1 previous similar message [ 4521.446831] Lustre: Skipped 2 previous similar messages [ 4531.679978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4531.682021] Lustre: Skipped 4 previous similar messages [ 4535.263081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4535.322260] Lustre: server umount lustre-MDT0000 complete [ 4536.519151] LustreError: 291416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425466 with bad export cookie 13017366279113079104 [ 4536.521196] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4536.523277] LustreError: 291416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4536.637568] Lustre: server umount lustre-MDT0001 complete [ 4540.013803] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4541.007544] Key type lgssc unregistered [ 4541.132374] LNet: 294139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4541.135331] LNetError: 294139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4541.147330] LNet: Removed LNI 192.168.201.114@tcp [ 4541.453856] Key type .llcrypt unregistered [ 4541.455076] Key type ._llcrypt unregistered [ 4544.909088] Key type ._llcrypt registered [ 4544.910605] Key type .llcrypt registered [ 4544.944417] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4545.240262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4545.249460] alg: No test for adler32 (adler32-zlib) [ 4546.102515] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 4546.182770] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4547.759171] Key type lgssc registered [ 4548.122324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4582.568633] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4585.491811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4585.497158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.587121] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4586.595361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4586.654464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.010701] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4590.934508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4590.969599] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4590.979572] Lustre: 296855:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4591.079403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4592.365464] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4594.178578] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4595.436119] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4608.751685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4608.830120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4608.897490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4610.796535] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4612.645356] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4613.403490] Lustre: server umount lustre-OST0000 complete [ 4615.666982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.668452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4615.671149] Lustre: Skipped 1 previous similar message [ 4620.758585] Lustre: server umount lustre-MDT0000 complete [ 4622.051580] LustreError: 296847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425552 with bad export cookie 17091649481532147400 [ 4622.055554] LustreError: 296847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4622.055732] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4622.190702] Lustre: server umount lustre-MDT0001 complete [ 4625.557727] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4626.469106] Key type lgssc unregistered [ 4626.592317] LNet: 300058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4626.594753] LNetError: 300058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4626.603272] LNet: Removed LNI 192.168.201.114@tcp [ 4626.902755] Key type .llcrypt unregistered [ 4626.904301] Key type ._llcrypt unregistered [ 4635.861513] Key type ._llcrypt registered [ 4635.863177] Key type .llcrypt registered [ 4635.905925] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4636.256042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4636.275102] alg: No test for adler32 (adler32-zlib) [ 4637.136594] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 4637.224417] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4638.807098] Key type lgssc registered [ 4639.173789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4641.833057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4641.840873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.996408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.173734] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4646.874731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.980709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4648.134112] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4649.767113] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4650.941245] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4653.191820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4653.310336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.216937] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4657.068922] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4677.601510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4677.604538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4677.609176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4678.112372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4678.114834] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.115335] Lustre: Skipped 1 previous similar message [ 4678.119061] Lustre: Skipped 1 previous similar message [ 4680.987832] Lustre: server umount lustre-MDT0000 complete [ 4682.188958] LustreError: 301415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425612 with bad export cookie 18271732543386837662 [ 4682.192263] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4682.195530] LustreError: 301415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4682.956851] LustreError: 303589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.962954] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 4682.964681] LustreError: 303589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4682.966745] Lustre: Skipped 1 previous similar message [ 4683.235650] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.263798] LustreError: 301431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.264055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4687.269097] LustreError: 301431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4687.270913] Lustre: Skipped 1 previous similar message [ 4688.372286] Lustre: server umount lustre-MDT0001 complete [ 4690.607438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4693.566611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.783645] LustreError: 300745:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000575ed5d6 x1863765791366528/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4706.912951] LustreError: 303039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.918615] LustreError: 303039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4706.940219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.964186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4708.313100] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4711.241815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.369547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4712.420578] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4712.700747] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4714.585516] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4717.544155] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4720.304889] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4721.569877] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4722.130864] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4726.752159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4726.754228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4726.758437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4726.760171] Lustre: Skipped 1 previous similar message [ 4727.778668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4729.621629] Lustre: server umount lustre-OST0000 complete [ 4731.871802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4731.874764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.879811] Lustre: Skipped 1 previous similar message [ 4738.016365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4738.019656] Lustre: Skipped 6 previous similar messages [ 4745.183194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4745.253217] Lustre: server umount lustre-MDT0000 complete [ 4746.600371] LustreError: 301417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425676 with bad export cookie 18271732543386844816 [ 4746.603916] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4746.604404] LustreError: 301417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4746.724261] Lustre: server umount lustre-MDT0001 complete [ 4750.344868] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4751.322234] Key type lgssc unregistered [ 4751.453397] LNet: 307144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4751.455820] LNetError: 307144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4751.468371] LNet: Removed LNI 192.168.201.114@tcp [ 4751.764835] Key type .llcrypt unregistered [ 4751.766048] Key type ._llcrypt unregistered [ 4760.859689] Key type ._llcrypt registered [ 4760.860774] Key type .llcrypt registered [ 4760.897996] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4761.323615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4761.341879] alg: No test for adler32 (adler32-zlib) [ 4762.197339] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 4762.282882] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4763.871105] Key type lgssc registered [ 4764.250207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4766.862816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4766.869626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4768.062474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.379604] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4772.226871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4772.356300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4773.651089] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4775.444632] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4776.668634] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4781.793096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.797176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4781.799086] Lustre: Skipped 1 previous similar message [ 4783.830306] Lustre: server umount lustre-MDT0000 complete [ 4785.079845] LustreError: 308503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425715 with bad export cookie 7992523676005045432 [ 4785.080632] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4785.085544] LustreError: 308503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4785.200988] Lustre: server umount lustre-MDT0001 complete [ 4790.032307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.130239] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4790.140667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4790.215282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.509682] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4794.284193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.322800] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4794.332177] Lustre: 310879:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4794.420900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4795.697326] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4797.500314] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4798.716405] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4801.251189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.320749] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4801.324678] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4801.327296] LustreError: 312536:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 4801.329456] LustreError: 312536:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4801.331303] LustreError: 312536:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4801.354474] Lustre: server umount lustre-OST0000 complete [ 4801.355842] LustreError: 312536:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4804.159644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4804.226645] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4804.310553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4806.244434] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4808.108195] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4809.712611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4810.694440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4810.725557] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4810.759602] Lustre: lustre-OST0001: new disk, initializing [ 4810.761481] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4810.764229] Lustre: 313965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4812.652351] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4814.454335] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4815.481326] Lustre: server umount lustre-OST0001 complete [ 4818.415478] LustreError: 313001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.417197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4818.418303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4818.421629] LustreError: 313001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4818.425016] Lustre: Skipped 2 previous similar messages [ 4823.525879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4823.527079] LustreError: 314853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.527851] Lustre: Skipped 2 previous similar messages [ 4823.532265] LustreError: 314853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4828.642055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4828.643305] LustreError: 313003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4828.643990] Lustre: Skipped 1 previous similar message [ 4828.649057] LustreError: 313003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4831.199081] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4831.250902] Lustre: server umount lustre-OST0000 complete [ 4833.761149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4833.762762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4833.765786] Lustre: Skipped 1 previous similar message [ 4833.769167] Lustre: Skipped 1 previous similar message [ 4838.616040] Lustre: server umount lustre-MDT0000 complete [ 4838.881289] LustreError: 310890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4838.886709] LustreError: 310890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4839.835134] LustreError: 310872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425770 with bad export cookie 7992523676005045978 [ 4839.838963] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4839.840761] LustreError: 310872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4839.958356] Lustre: server umount lustre-MDT0001 complete [ 4843.566451] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4844.570907] Key type lgssc unregistered [ 4844.701303] LNet: 315789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4844.703976] LNetError: 315789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4844.710413] LNet: Removed LNI 192.168.201.114@tcp [ 4845.007876] Key type .llcrypt unregistered [ 4845.009080] Key type ._llcrypt unregistered [ 4875.613103] Key type ._llcrypt registered [ 4875.614883] Key type .llcrypt registered [ 4875.662546] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 4881.669677] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4882.268643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4882.290074] alg: No test for adler32 (adler32-zlib) [ 4883.151255] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 4883.242064] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4884.839201] Key type lgssc registered [ 4885.238557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4896.659743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4896.666200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4897.765275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4897.778092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4897.815554] Lustre: lustre-MDT0000: new disk, initializing [ 4897.840884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4897.848600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4899.217262] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4903.310762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.353865] Lustre: 320667:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4903.378527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4903.380416] Lustre: Skipped 1 previous similar message [ 4903.414881] Lustre: lustre-MDT0001: new disk, initializing [ 4903.447358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4903.458050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4903.461454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4904.714096] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4907.779647] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4908.966442] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4911.280838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4911.394169] Lustre: lustre-OST0000: new disk, initializing [ 4911.396119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4911.399595] Lustre: 322409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4911.430060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.350230] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4914.487846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4914.492215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4914.522974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4916.508966] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4917.958132] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4918.049870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4919.564161] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4919.661105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4922.387973] Lustre: server umount lustre-OST0000 complete [ 4923.871844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4923.875203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4923.880768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4924.897483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4924.898464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.016162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4930.018606] Lustre: Skipped 2 previous similar messages [ 4934.138988] Lustre: server umount lustre-MDT0000 complete [ 4935.428109] LustreError: 321444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425865 with bad export cookie 7877254446093515838 [ 4935.431621] LustreError: 321444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4935.431689] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4944.915590] Lustre: server umount lustre-MDT0001 complete [ 4947.843355] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:24:37 (1777425877) [ 4950.527543] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:24:40 (1777425880) [ 4953.481813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4953.666828] LustreError: 325016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4953.672112] LustreError: 325016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4953.693950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.031490] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4957.942637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4958.075468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4959.514236] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4961.393873] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4962.621187] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4964.936451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4967.014762] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4968.815483] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4987.979868] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4988.593459] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4989.151793] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4989.698599] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4990.373170] Lustre: Failing over lustre-MDT0000 [ 4990.527418] Lustre: server umount lustre-MDT0000 complete [ 4994.016026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4994.018462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.023986] LustreError: 325034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.042187] LustreError: 325034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.047154] LustreError: 325034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4999.885262] LustreError: 326505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.448365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.493897] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5003.590385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5003.593149] Lustre: Skipped 1 previous similar message [ 5003.617209] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5004.970300] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5005.006167] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5008.873418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5008.883805] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5008.899660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5010.121123] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5010.678349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5012.575846] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5013.254080] Lustre: Failing over lustre-MDT0000 [ 5013.426794] Lustre: server umount lustre-MDT0000 complete [ 5013.987375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.987984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5013.991302] Lustre: Skipped 4 previous similar messages [ 5013.993888] LustreError: 326505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.998637] LustreError: 326505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5024.225176] LustreError: 325010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.230685] LustreError: 325010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5026.309615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.348345] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5026.439327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5026.466448] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5027.729314] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5030.606130] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5031.911984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5031.913800] Lustre: Skipped 2 previous similar messages [ 5031.923533] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5031.938970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5033.145730] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5033.750661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5037.024859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.025792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5037.026991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5037.028204] Lustre: Skipped 1 previous similar message [ 5042.145186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5042.147199] Lustre: Skipped 4 previous similar messages [ 5042.204180] Lustre: server umount lustre-MDT0000 complete [ 5043.402948] LustreError: 324995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777425973 with bad export cookie 7877254446093518491 [ 5043.405909] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.406660] LustreError: 324995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5043.511737] Lustre: server umount lustre-MDT0001 complete [ 5055.075586] Lustre: server umount lustre-OST0000 complete [ 5060.284636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5060.442280] LustreError: 330813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5060.448022] LustreError: 330813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5060.466539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.818253] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5064.870562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.397334] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5068.284862] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5069.498782] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5071.799131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5073.917983] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5075.734146] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5085.286880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5090.785856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5090.788596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.791886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5090.793389] Lustre: Skipped 2 previous similar messages [ 5093.708907] Lustre: server umount lustre-OST0000 complete [ 5109.215178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5109.269851] Lustre: server umount lustre-MDT0000 complete [ 5110.506401] LustreError: 332976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426040 with bad export cookie 7877254446093519233 [ 5110.509230] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5110.510031] LustreError: 332976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5110.629794] Lustre: server umount lustre-MDT0001 complete [ 5114.353689] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5115.442322] Key type lgssc unregistered [ 5115.571500] LNet: 334012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5115.574964] LNetError: 334012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5115.583327] LNet: Removed LNI 192.168.201.114@tcp [ 5115.906255] Key type .llcrypt unregistered [ 5115.907719] Key type ._llcrypt unregistered [ 5125.862962] Key type ._llcrypt registered [ 5125.864433] Key type .llcrypt registered [ 5125.901553] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5126.305492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5126.331961] alg: No test for adler32 (adler32-zlib) [ 5127.183963] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 5127.265983] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5128.847185] Key type lgssc registered [ 5129.241541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5131.808678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5131.814693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.965951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.221884] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5137.090937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5137.206147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.522648] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5140.321326] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5141.496850] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5143.740106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5143.847705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.815215] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5147.605391] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5158.184070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5168.095813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5168.098094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.101947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5168.609045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.613373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5168.615290] Lustre: Skipped 1 previous similar message [ 5173.277777] Lustre: server umount lustre-MDT0000 complete [ 5173.728103] LustreError: 335489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5173.733217] LustreError: 335489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5174.525274] LustreError: 335470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426104 with bad export cookie 6187148470592584625 [ 5174.529227] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5174.530691] LustreError: 335470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5174.654684] Lustre: server umount lustre-MDT0001 complete [ 5186.262647] Lustre: server umount lustre-OST0000 complete [ 5190.610154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.761047] LustreError: 338882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.782864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5192.038099] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5194.928684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5195.063544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5196.371708] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5198.194640] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5199.387114] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5201.634857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5203.620162] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5205.323565] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5216.103304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5231.071969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5231.074330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.077710] Lustre: Skipped 1 previous similar message [ 5231.079512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5234.137030] Lustre: server umount lustre-MDT0000 complete [ 5235.445052] LustreError: 338863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426165 with bad export cookie 6187148470592585724 [ 5235.447754] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5235.448689] LustreError: 338863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5235.575113] Lustre: server umount lustre-MDT0001 complete [ 5247.134908] Lustre: server umount lustre-OST0000 complete [ 5252.375318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.551605] LustreError: 342476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5252.577276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5252.579513] Lustre: Skipped 1 previous similar message [ 5253.893350] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5256.765380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.262106] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5260.056226] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5261.288975] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5263.599090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.611894] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5267.444414] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5277.035860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5282.276701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5282.277395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.278926] LustreError: Skipped 1 previous similar message [ 5282.279597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5282.279603] Lustre: Skipped 2 previous similar messages [ 5282.284153] Lustre: Skipped 3 previous similar messages [ 5285.208070] Lustre: server umount lustre-OST0000 complete [ 5287.394216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.395775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.398848] Lustre: Skipped 1 previous similar message [ 5287.402038] Lustre: Skipped 2 previous similar messages [ 5292.633272] Lustre: server umount lustre-MDT0000 complete [ 5293.869750] LustreError: 343193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426224 with bad export cookie 6187148470592586732 [ 5293.873774] LustreError: 343193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5293.873887] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.002446] Lustre: server umount lustre-MDT0001 complete [ 5297.515456] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5298.594572] Key type lgssc unregistered [ 5298.721513] LNet: 345673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5298.724728] LNetError: 345673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5298.734448] LNet: Removed LNI 192.168.201.114@tcp [ 5299.054362] Key type .llcrypt unregistered [ 5299.055256] Key type ._llcrypt unregistered [ 5308.639602] Key type ._llcrypt registered [ 5308.640622] Key type .llcrypt registered [ 5308.684825] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5309.232295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5309.244725] alg: No test for adler32 (adler32-zlib) [ 5310.101476] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 5310.196387] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5311.783144] Key type lgssc registered [ 5312.143606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5314.827886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5314.835675] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5314.839485] LustreError: 347073:0:(osd_handler.c:8556:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5314.849443] LustreError: 347073:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5314.852676] LustreError: 347073:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5314.855274] LustreError: 347073:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5314.858458] LustreError: 347073:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5317.483818] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5317.486089] LustreError: 347471:0:(osd_handler.c:8556:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5317.494088] LustreError: 347471:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5317.496733] LustreError: 347471:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5317.498649] LustreError: 347471:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5317.502569] LustreError: 347471:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5324.121081] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5325.253900] Key type lgssc unregistered [ 5325.396394] LNet: 348454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5325.399091] LNetError: 348454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5325.407304] LNet: Removed LNI 192.168.201.114@tcp [ 5325.736436] Key type .llcrypt unregistered [ 5325.737725] Key type ._llcrypt unregistered [ 5331.918288] Key type ._llcrypt registered [ 5331.919326] Key type .llcrypt registered [ 5331.964581] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 5338.403867] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5338.794523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5338.827064] alg: No test for adler32 (adler32-zlib) [ 5339.684086] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 5339.773063] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5341.359106] Key type lgssc registered [ 5341.785933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5353.658108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5353.663827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5354.763554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5354.777316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5354.815367] Lustre: lustre-MDT0000: new disk, initializing [ 5354.840039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5354.847095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5356.158311] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5360.295218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.336913] Lustre: 351551:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5360.358574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5360.360412] Lustre: Skipped 1 previous similar message [ 5360.396506] Lustre: lustre-MDT0001: new disk, initializing [ 5360.428116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5360.440075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5360.444384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5361.689895] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5364.761445] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5365.994768] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5368.321147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5368.436254] Lustre: lustre-OST0000: new disk, initializing [ 5368.438361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5368.441042] Lustre: 353294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5368.477526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5370.457574] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5371.961942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5371.966392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5371.999958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5373.589526] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5375.012849] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5375.091526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5376.369106] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5376.436205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5382.114283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.114588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5382.115743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5382.118513] Lustre: Skipped 1 previous similar message [ 5382.120752] LustreError: Skipped 1 previous similar message [ 5387.233370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5387.235855] Lustre: Skipped 2 previous similar messages [ 5391.327146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5391.389622] Lustre: server umount lustre-OST0000 complete [ 5396.448042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5396.451851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.457981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5397.473039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.746850] Lustre: server umount lustre-MDT0000 complete [ 5399.967590] LustreError: 352329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426330 with bad export cookie 1474063016307692186 [ 5399.970232] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5399.973740] LustreError: 352329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5400.106749] Lustre: server umount lustre-MDT0001 complete [ 5402.829947] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5403.370564] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:32:13 (1777426333) [ 5405.490204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.673190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.963051] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5409.713531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.836920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5411.107120] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5412.886495] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5414.047145] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5416.343981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5418.363572] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5420.594901] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5422.934510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.009736] Lustre: lustre-OST0001: new disk, initializing [ 5423.011756] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5423.014203] Lustre: 358101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5423.055295] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5423.056887] Lustre: Skipped 1 previous similar message [ 5425.230774] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5425.669817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5425.673332] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5425.708236] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5428.413471] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5430.752959] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5430.753346] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.755370] LustreError: Skipped 1 previous similar message [ 5430.756519] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5430.760344] Lustre: Skipped 1 previous similar message [ 5430.761718] Lustre: Skipped 2 previous similar messages [ 5435.101521] Lustre: lustre-OST0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 5435.104620] Lustre: Skipped 2 previous similar messages [ 5436.350376] Lustre: server umount lustre-OST0001 complete [ 5436.900627] LustreError: 357199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.905124] LustreError: 357199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5442.020050] LustreError: 358796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.024403] LustreError: 358796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5447.139324] LustreError: 357741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5447.143949] LustreError: 357741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5451.232210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5451.234492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.240682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5452.263529] LustreError: 358796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.271874] LustreError: 358796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5454.480377] Lustre: server umount lustre-OST0000 complete [ 5455.839826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5455.843080] LustreError: Skipped 1 previous similar message [ 5467.616857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5467.619680] Lustre: Skipped 8 previous similar messages [ 5470.175122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5470.240894] Lustre: server umount lustre-MDT0000 complete [ 5471.449579] LustreError: 355630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426401 with bad export cookie 1474063016307693054 [ 5471.451750] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5471.453154] LustreError: 355630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5477.618440] Lustre: server umount lustre-MDT0001 complete [ 5481.114145] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5482.196864] Key type lgssc unregistered [ 5482.332523] LNet: 359957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5482.335654] LNetError: 359957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5482.346369] LNet: Removed LNI 192.168.201.114@tcp [ 5482.652371] Key type .llcrypt unregistered [ 5482.653679] Key type ._llcrypt unregistered [ 5500.880252] Key type ._llcrypt registered [ 5500.881327] Key type .llcrypt registered [ 5500.917069] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5501.275820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5501.313898] alg: No test for adler32 (adler32-zlib) [ 5502.173053] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 5502.257635] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5503.839180] Key type lgssc registered [ 5504.223613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5506.893637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5506.899722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.006686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5508.014301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5508.083368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.409815] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5512.120082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5512.152455] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5512.164323] Lustre: 362714:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5512.261463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5513.508979] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5515.531382] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5516.806524] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5521.889905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5521.890989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5521.896601] Lustre: Skipped 1 previous similar message [ 5523.803151] Lustre: server umount lustre-MDT0000 complete [ 5525.093604] LustreError: 364080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426455 with bad export cookie 1069276044974972042 [ 5525.097316] LustreError: 364080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5525.097857] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.220628] Lustre: server umount lustre-MDT0001 complete [ 5528.741215] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:34:18 (1777426458) [ 5531.333717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.525284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.771247] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5536.316761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.434480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5537.704143] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5539.606780] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5540.838795] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5543.090421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5543.162610] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5543.228698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.059552] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5546.943368] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5550.735713] Lustre: Permanently deactivating lustre-OST0000 [ 5550.746099] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5555.197377] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5555.713940] LustreError: 367592:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5555.718770] LustreError: 367592:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5555.721140] LustreError: 367592:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5556.344436] LustreError: 367641:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5556.346489] LustreError: 367641:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5563.342182] Lustre: server umount lustre-OST0000 complete [ 5564.209724] LustreError: 367891:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5564.212134] LustreError: 367891:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5565.920712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.921133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5565.924122] Lustre: Skipped 1 previous similar message [ 5565.925936] Lustre: Skipped 1 previous similar message [ 5571.040586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5571.042784] Lustre: Skipped 2 previous similar messages [ 5576.159928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5576.161785] Lustre: Skipped 1 previous similar message [ 5579.231146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5579.288686] Lustre: server umount lustre-MDT0000 complete [ 5580.567241] LustreError: 364980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426510 with bad export cookie 1069276044974972609 [ 5580.570218] LustreError: 364980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5580.570613] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5590.034050] Lustre: server umount lustre-MDT0001 complete [ 5592.844073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5594.182279] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5595.119887] LustreError: 369033:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5597.488644] Lustre: 369371:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.114@tcp [ 5598.240162] Lustre: server umount MGS complete [ 5602.837254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.027371] LustreError: 370095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5603.032173] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5603.033364] LustreError: 370095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5603.035953] Lustre: Skipped 1 previous similar message [ 5603.058652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.381592] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5607.649911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5607.759953] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5609.209705] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5611.082927] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5612.373908] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5613.038719] Lustre: Permanently reactivating lustre-OST0000 [ 5613.045187] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5613.047185] Lustre: Skipped 2 previous similar messages [ 5636.063164] LustreError: 371567:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5638.916822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5639.055246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5639.057866] Lustre: Skipped 1 previous similar message [ 5640.880585] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5642.957683] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5656.545266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.546090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5656.550803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5656.552043] LustreError: Skipped 1 previous similar message [ 5656.553597] Lustre: Skipped 1 previous similar message [ 5661.134970] Lustre: server umount lustre-OST0000 complete [ 5664.223960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5664.227633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.231035] Lustre: Skipped 1 previous similar message [ 5666.784611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5666.786458] Lustre: Skipped 2 previous similar messages [ 5668.503992] Lustre: server umount lustre-MDT0000 complete [ 5669.730262] LustreError: 370075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426600 with bad export cookie 1069276044974973645 [ 5669.733290] LustreError: 370075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5669.734968] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.863621] Lustre: server umount lustre-MDT0001 complete [ 5673.404206] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5674.382436] Key type lgssc unregistered [ 5674.502381] LNet: 374266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5674.504623] LNetError: 374266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5674.511467] LNet: Removed LNI 192.168.201.114@tcp [ 5674.798505] Key type .llcrypt unregistered [ 5674.799900] Key type ._llcrypt unregistered [ 5681.003776] Key type ._llcrypt registered [ 5681.004893] Key type .llcrypt registered [ 5681.043548] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 5687.293983] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5687.700046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5687.732264] alg: No test for adler32 (adler32-zlib) [ 5688.611990] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 5688.713938] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5690.303249] Key type lgssc registered [ 5690.730243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5701.975315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5701.980779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.087304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5703.100106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5703.142190] Lustre: lustre-MDT0000: new disk, initializing [ 5703.168289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.175091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5704.452805] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5708.588737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.631463] Lustre: 377362:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5708.662122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5708.664604] Lustre: Skipped 1 previous similar message [ 5708.705442] Lustre: lustre-MDT0001: new disk, initializing [ 5708.738325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5708.752726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5708.755825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5710.092600] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5713.308337] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5714.618901] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5716.868024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5716.975962] Lustre: lustre-OST0000: new disk, initializing [ 5716.978061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5716.981069] Lustre: 379105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5717.015126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.900425] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5721.869082] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5723.219805] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5727.225487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5727.228051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5727.260543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5727.389476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 5728.728505] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5728.822053] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5732.324977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5732.326641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5732.328195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5732.330825] Lustre: Skipped 1 previous similar message [ 5732.335097] LustreError: Skipped 1 previous similar message [ 5735.635187] Lustre: server umount lustre-OST0000 complete [ 5737.442300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.443427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5737.449670] Lustre: Skipped 1 previous similar message [ 5737.453489] Lustre: Skipped 1 previous similar message [ 5742.560553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5742.563795] Lustre: Skipped 2 previous similar messages [ 5747.680628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5747.684182] Lustre: Skipped 1 previous similar message [ 5751.263099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5751.333966] Lustre: server umount lustre-MDT0000 complete [ 5752.559084] LustreError: 378140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426682 with bad export cookie 11343913084561384031 [ 5752.562234] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5752.562496] LustreError: 378140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5752.679057] Lustre: server umount lustre-MDT0001 complete [ 5755.444249] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5756.033392] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:38:05 (1777426685) [ 5758.221050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5758.394245] LustreError: 381443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.399023] LustreError: 381443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5758.422824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.769039] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5762.707266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5762.849697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5764.186455] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5765.995920] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5767.204532] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5769.561593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5771.670527] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5773.413649] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5774.107848] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5784.857633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5788.133545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.133587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5788.137417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5788.138885] LustreError: Skipped 1 previous similar message [ 5788.140800] Lustre: Skipped 1 previous similar message [ 5792.404577] Lustre: server umount lustre-OST0000 complete [ 5793.760168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5793.763145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.767864] Lustre: Skipped 1 previous similar message [ 5798.368504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5798.369146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5798.370526] Lustre: Skipped 1 previous similar message [ 5808.095096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5808.157929] Lustre: server umount lustre-MDT0000 complete [ 5808.608373] LustreError: 381443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5808.618712] LustreError: 381443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5809.423184] LustreError: 381425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426739 with bad export cookie 11343913084561384899 [ 5809.426411] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5809.426927] LustreError: 381425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5809.541334] Lustre: server umount lustre-MDT0001 complete [ 5813.331497] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5814.385672] Key type lgssc unregistered [ 5814.520470] LNet: 384834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5814.524632] LNetError: 384834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5814.536459] LNet: Removed LNI 192.168.201.114@tcp [ 5814.836990] Key type .llcrypt unregistered [ 5814.837950] Key type ._llcrypt unregistered [ 5822.665157] Key type ._llcrypt registered [ 5822.666397] Key type .llcrypt registered [ 5822.711193] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5822.840530] Key type .llcrypt unregistered [ 5822.841525] Key type ._llcrypt unregistered [ 5829.905701] Key type ._llcrypt registered [ 5829.906830] Key type .llcrypt registered [ 5829.950933] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5830.342461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5830.385102] alg: No test for adler32 (adler32-zlib) [ 5831.240650] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 5831.342306] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5832.927195] Key type lgssc registered [ 5833.332977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5836.102852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5836.109282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.270139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5838.561649] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5841.696571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5841.822102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.968851] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5845.906795] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5846.944786] LustreError: 386988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.951064] LustreError: 386988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5846.951905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5848.356673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5848.495427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5849.753056] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5858.788749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.791306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.803779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.808777] Lustre: Skipped 1 previous similar message [ 5863.511886] Lustre: server umount lustre-OST0000 complete [ 5868.001580] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5868.003419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.006254] Lustre: Skipped 1 previous similar message [ 5868.007930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.024161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.024856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.144917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5874.146805] Lustre: Skipped 2 previous similar messages [ 5879.263121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5879.264570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5879.269518] Lustre: Skipped 1 previous similar message [ 5879.337157] Lustre: server umount lustre-MDT0000 complete [ 5880.587566] LustreError: 386969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426810 with bad export cookie 12529281423359254979 [ 5880.590452] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5880.590842] LustreError: 386969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5880.717926] Lustre: server umount lustre-MDT0001 complete [ 5884.430398] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5885.687769] Key type lgssc unregistered [ 5885.821765] LNet: 389977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5885.826168] LNetError: 389977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5885.836694] LNet: Removed LNI 192.168.201.114@tcp [ 5886.306505] Key type .llcrypt unregistered [ 5886.307930] Key type ._llcrypt unregistered [ 5896.918067] Key type ._llcrypt registered [ 5896.919649] Key type .llcrypt registered [ 5896.962210] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5897.481897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5897.497918] alg: No test for adler32 (adler32-zlib) [ 5898.366422] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 5898.451933] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5900.039164] Key type lgssc registered [ 5900.397996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5903.466972] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5903.479724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5918.943447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5924.063263] LustreError: 391328:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.114@tcp: failed processing log, type 4: rc = -110 [ 5950.687203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.889107] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5954.954077] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5957.423861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5957.626077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5957.663088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5958.953600] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5961.949534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5962.069192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5963.308704] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5965.095404] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5966.338695] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5970.404447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.411227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5970.415527] Lustre: Skipped 1 previous similar message [ 5974.762627] Lustre: server umount lustre-OST0000 complete [ 5977.568235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5977.572046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.580443] Lustre: Skipped 1 previous similar message [ 5977.583929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.641097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.642063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.498438] Lustre: server umount lustre-MDT0000 complete [ 5983.843266] LustreError: 393521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426914 with bad export cookie 18446288386738289008 [ 5983.847560] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5983.966510] Lustre: server umount lustre-MDT0001 complete [ 5987.489516] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5988.880694] Key type lgssc unregistered [ 5989.040706] LNet: 394542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5989.045650] LNetError: 394542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5989.054681] LNet: Removed LNI 192.168.201.114@tcp [ 5989.458426] Key type .llcrypt unregistered [ 5989.460225] Key type ._llcrypt unregistered [ 5998.168544] Key type ._llcrypt registered [ 5998.169750] Key type .llcrypt registered [ 5998.205098] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5998.536222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5998.547250] alg: No test for adler32 (adler32-zlib) [ 5999.400282] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 5999.483275] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6001.071279] Key type lgssc registered [ 6001.500075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6004.562405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6004.571975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.745127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6007.256446] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6010.725734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.928117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6012.383957] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6014.269831] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6015.674804] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6017.909980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6018.022303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.801125] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6021.627180] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6032.360640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6041.568292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6041.570597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.577150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6042.325373] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 6042.593975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.597897] Lustre: Skipped 1 previous similar message [ 6044.186645] Lustre: server umount lustre-MDT0000 complete [ 6044.948421] LustreError: 395917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.14@tcp arrived at 1777426975 with bad export cookie 17625083207768802260 [ 6044.953206] LustreError: 395917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6045.586294] LustreError: 395917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777426975 with bad export cookie 17625083207768801609 [ 6045.590894] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6046.688871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6046.693730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6046.702132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6046.705256] Lustre: Skipped 3 previous similar messages [ 6047.711979] LustreError: 395936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.716396] LustreError: 395936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6050.720303] LustreError: 395936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.720799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6050.729694] LustreError: 395936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6050.737819] Lustre: Skipped 1 previous similar message [ 6051.737981] Lustre: server umount lustre-OST0000 complete [ 6054.465615] Lustre: server umount lustre-MDT0001 complete [ 6058.732664] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6060.281219] Key type lgssc unregistered [ 6060.465731] LNet: 399288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6060.471789] LNetError: 399288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6060.483596] LNet: Removed LNI 192.168.201.114@tcp [ 6060.926916] Key type .llcrypt unregistered [ 6060.929288] Key type ._llcrypt unregistered [ 6071.400952] Key type ._llcrypt registered [ 6071.401916] Key type .llcrypt registered [ 6071.447814] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6071.965441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6071.977371] alg: No test for adler32 (adler32-zlib) [ 6072.837054] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 6072.930966] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6074.519152] Key type lgssc registered [ 6075.050044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6077.969425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6077.978320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.176076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.496111] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6083.680972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6083.818425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6085.128113] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6086.899665] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6088.356720] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6090.915575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6091.102448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.119647] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6095.507460] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6104.428300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6109.666076] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6109.666208] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6109.669939] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6109.673369] Lustre: Skipped 1 previous similar message [ 6111.444140] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 6113.585703] Lustre: server umount lustre-MDT0001 complete [ 6114.785456] LustreError: 401389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.790730] LustreError: 401389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6119.904432] LustreError: 401389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.904960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6119.905263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6119.905270] Lustre: Skipped 1 previous similar message [ 6119.906077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6121.548680] Lustre: server umount lustre-OST0000 complete [ 6123.060384] Lustre: server umount lustre-MDT0000 complete [ 6127.605703] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6128.598072] Key type lgssc unregistered [ 6128.721357] LNet: 404016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6128.723670] LNetError: 404016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6128.731319] LNet: Removed LNI 192.168.201.114@tcp [ 6129.013403] Key type .llcrypt unregistered [ 6129.014564] Key type ._llcrypt unregistered [ 6136.612207] Key type ._llcrypt registered [ 6136.613450] Key type .llcrypt registered [ 6136.651293] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 6142.561662] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6142.926623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6142.960659] alg: No test for adler32 (adler32-zlib) [ 6143.802941] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 6143.889211] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6145.463153] Key type lgssc registered [ 6145.815802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6157.605112] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6164.530759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6164.537321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6165.630222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6165.641055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6165.679578] Lustre: lustre-MDT0000: new disk, initializing [ 6165.704146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6165.710430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6166.977202] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6172.921315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.032313] Lustre: lustre-OST0000: new disk, initializing [ 6173.034471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6173.036368] Lustre: Skipped 1 previous similar message [ 6173.038729] Lustre: 408952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6173.065956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6173.823246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6173.826172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6173.835423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6174.950535] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6178.501298] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6184.396667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.442028] Lustre: 407951:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6184.470249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6184.509564] Lustre: lustre-MDT0001: new disk, initializing [ 6184.549979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6184.562052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6184.565285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6185.880789] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6190.119872] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6190.247368] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6191.922887] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6192.041888] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6196.444365] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 6196.707213] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6196.709316] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.489780] Lustre: server umount lustre-MDT0001 complete [ 6206.947829] LustreError: 408960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6206.953410] LustreError: 408960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6212.066083] LustreError: 411526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6212.072844] LustreError: 411526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6215.136473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6215.143166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.152282] Lustre: Skipped 1 previous similar message [ 6215.156979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6215.160563] Lustre: Skipped 2 previous similar messages [ 6217.184716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.185292] LustreError: 410181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6219.153208] Lustre: server umount lustre-OST0000 complete [ 6226.809788] Lustre: server umount lustre-MDT0000 complete [ 6231.938158] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6232.995550] Key type lgssc unregistered [ 6233.122612] LNet: 412520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6233.125400] LNetError: 412520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6233.134301] LNet: Removed LNI 192.168.201.114@tcp [ 6233.443944] Key type .llcrypt unregistered [ 6233.445902] Key type ._llcrypt unregistered [ 6247.985865] Key type ._llcrypt registered [ 6247.986921] Key type .llcrypt registered [ 6248.033090] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6248.153277] Key type .llcrypt unregistered [ 6248.155143] Key type ._llcrypt unregistered [ 6255.355424] Key type ._llcrypt registered [ 6255.356571] Key type .llcrypt registered [ 6255.408762] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6255.810554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6255.838054] alg: No test for adler32 (adler32-zlib) [ 6256.757735] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 6256.871343] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6258.463193] Key type lgssc registered [ 6258.840996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6268.218804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6268.224884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.323892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6269.336201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6269.373606] Lustre: lustre-MDT0000: new disk, initializing [ 6269.401862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6269.410136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6270.776713] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6275.171340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.215282] Lustre: 415844:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6275.240393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6275.242729] Lustre: Skipped 1 previous similar message [ 6275.281632] Lustre: lustre-MDT0001: new disk, initializing [ 6275.313488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6275.327128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6275.330083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6276.602775] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6279.927902] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6281.447365] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6283.986927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6284.107216] Lustre: lustre-OST0000: new disk, initializing [ 6284.109128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6284.112383] Lustre: 417586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6284.139840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.163932] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6286.401905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6286.406419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6286.451683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6289.643400] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6296.031806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6296.034080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6296.037555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.057473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.058509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.060673] Lustre: Skipped 1 previous similar message [ 6297.064462] Lustre: Skipped 1 previous similar message [ 6298.332569] Lustre: server umount lustre-MDT0000 complete [ 6299.635485] LustreError: 415833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427229 with bad export cookie 4669177857209108997 [ 6299.640602] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6299.755934] Lustre: server umount lustre-MDT0001 complete [ 6311.122989] Lustre: server umount lustre-OST0000 complete [ 6314.499998] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:47:24 (1777427244) [ 6317.425388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.658637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6319.068170] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6322.548361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.678371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6324.146662] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6326.280194] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6327.776428] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6341.164426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6341.277203] Lustre: Found index 0 for lustre-OST0000, updating log [ 6341.361622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.386176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6343.738869] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6345.889698] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6347.747473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6347.748730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.758152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6349.278428] Lustre: server umount lustre-OST0000 complete [ 6363.616092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6363.618434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.622994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.627423] Lustre: Skipped 2 previous similar messages [ 6368.026928] Lustre: server umount lustre-MDT0000 complete [ 6369.349762] LustreError: 419477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427299 with bad export cookie 4669177857209110376 [ 6369.353439] LustreError: 419477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6369.355378] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6369.507976] Lustre: server umount lustre-MDT0001 complete [ 6372.397614] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:48:22 (1777427302) [ 6405.343359] LNet: 1 local NIs in recovery (showing 1): 192.168.201.114@tcp [ 6955.858375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6956.069747] LustreError: 425559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6956.079746] LustreError: 425559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6956.101584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6957.728097] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6961.093451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6961.119975] LustreError: 425559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.443541] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6964.634554] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6966.210558] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6980.414647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6980.550594] Lustre: Found index 0 for lustre-OST0000, updating log [ 6983.227177] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6983.783909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6985.225227] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6987.169947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.173591] Lustre: Skipped 1 previous similar message [ 6987.175889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6987.178307] Lustre: Skipped 2 previous similar messages [ 6988.197617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6992.313456] Lustre: server umount lustre-OST0000 complete [ 6994.047979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.049163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.055760] Lustre: Skipped 1 previous similar message [ 6996.129210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6996.132846] Lustre: Skipped 4 previous similar messages [ 7000.224964] LustreError: 426281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.233912] LustreError: 426281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7000.241047] Lustre: server umount lustre-MDT0000 complete [ 7001.823654] LustreError: 425540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777427932 with bad export cookie 4669177857209111433 [ 7001.828101] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7001.829155] LustreError: 425540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7001.963102] Lustre: server umount lustre-MDT0001 complete [ 7019.744348] LNet: 1 local NIs in recovery (showing 1): 192.168.201.114@tcp [ 7021.649031] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 7026.231286] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7043.026123] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:59:32 (1777427972) [ 7051.124088] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7052.617398] Key type lgssc unregistered [ 7052.783951] LNet: 431824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7052.788490] LNetError: 431824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7112.229538] LNet: Removed LNI 192.168.201.114@tcp [ 7112.677428] Key type .llcrypt unregistered [ 7112.679723] Key type ._llcrypt unregistered [ 7120.387164] Key type ._llcrypt registered [ 7120.388646] Key type .llcrypt registered [ 7120.434915] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7120.784038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7120.823636] alg: No test for adler32 (adler32-zlib) [ 7121.679791] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 7121.781878] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7123.383154] Key type lgssc registered [ 7123.909934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7125.951839] LNet: Added LNI 192.168.201.114@tcp100 [8/256/0/180] [ 7126.373487] LNet: Added LNI 192.168.201.114@tcp101 [8/256/0/180] [ 7126.797467] LNet: Added LNI 192.168.201.114@tcp102 [8/256/0/180] [ 7127.224669] LNet: Added LNI 192.168.201.114@tcp103 [8/256/0/180] [ 7127.644112] LNet: Added LNI 192.168.201.114@tcp104 [8/256/0/180] [ 7128.062855] LNet: Added LNI 192.168.201.114@tcp105 [8/256/0/180] [ 7128.494941] LNet: Added LNI 192.168.201.114@tcp106 [8/256/0/180] [ 7128.928952] LNet: Added LNI 192.168.201.114@tcp107 [8/256/0/180] [ 7129.348797] LNet: Added LNI 192.168.201.114@tcp108 [8/256/0/180] [ 7129.749588] LNet: Added LNI 192.168.201.114@tcp109 [8/256/0/180] [ 7130.182745] LNet: Added LNI 192.168.201.114@tcp110 [8/256/0/180] [ 7130.606886] LNet: Added LNI 192.168.201.114@tcp111 [8/256/0/180] [ 7131.032808] LNet: Added LNI 192.168.201.114@tcp112 [8/256/0/180] [ 7131.459988] LNet: Added LNI 192.168.201.114@tcp113 [8/256/0/180] [ 7131.877883] LNet: Added LNI 192.168.201.114@tcp114 [8/256/0/180] [ 7132.310972] LNet: Added LNI 192.168.201.114@tcp115 [8/256/0/180] [ 7132.738856] LNet: Added LNI 192.168.201.114@tcp116 [8/256/0/180] [ 7133.158860] LNet: Added LNI 192.168.201.114@tcp117 [8/256/0/180] [ 7133.557732] LNet: Added LNI 192.168.201.114@tcp118 [8/256/0/180] [ 7133.985487] LNet: Added LNI 192.168.201.114@tcp119 [8/256/0/180] [ 7134.413522] LNet: Added LNI 192.168.201.114@tcp120 [8/256/0/180] [ 7134.837617] LNet: Added LNI 192.168.201.114@tcp121 [8/256/0/180] [ 7135.269685] LNet: Added LNI 192.168.201.114@tcp122 [8/256/0/180] [ 7135.687169] LNet: Added LNI 192.168.201.114@tcp123 [8/256/0/180] [ 7136.109863] LNet: Added LNI 192.168.201.114@tcp124 [8/256/0/180] [ 7136.516921] LNet: Added LNI 192.168.201.114@tcp125 [8/256/0/180] [ 7136.940923] LNet: Added LNI 192.168.201.114@tcp126 [8/256/0/180] [ 7137.375622] LNet: Added LNI 192.168.201.114@tcp127 [8/256/0/180] [ 7137.793820] LNet: Added LNI 192.168.201.114@tcp128 [8/256/0/180] [ 7138.219391] LNet: Added LNI 192.168.201.114@tcp129 [8/256/0/180] [ 7138.650847] LNet: Added LNI 192.168.201.114@tcp130 [8/256/0/180] [ 7139.021507] LNet: Added LNI 192.168.201.114@tcp131 [8/256/0/180] [ 7139.428643] LNet: Added LNI 192.168.201.114@tcp132 [8/256/0/180] [ 7139.858361] LNet: Added LNI 192.168.201.114@tcp133 [8/256/0/180] [ 7140.286235] LNet: Added LNI 192.168.201.114@tcp134 [8/256/0/180] [ 7140.708973] LNet: Added LNI 192.168.201.114@tcp135 [8/256/0/180] [ 7143.978429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7143.989677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.066922] Lustre: 434817:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [ 7145.145364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7145.168934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7145.225573] Lustre: lustre-MDT0000: new disk, initializing [ 7145.270552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.281954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7147.103360] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7152.651596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7152.700476] Lustre: 434834:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7152.730603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7152.733391] Lustre: Skipped 1 previous similar message [ 7152.777136] Lustre: lustre-MDT0001: new disk, initializing [ 7152.790073] Lustre: 435615:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [ 7152.794487] Lustre: 435615:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7152.845977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7152.867253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7152.876644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7154.510545] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7158.535588] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7160.277920] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7163.508536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7163.660601] Lustre: lustre-OST0000: new disk, initializing [ 7163.664218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7163.668947] Lustre: 436576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7163.717615] Lustre: 436581:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [ 7163.723761] Lustre: 436581:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7163.751874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7166.160003] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7169.959824] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7171.581678] Lustre: 437274:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [ 7171.586096] Lustre: 437274:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7171.609455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7171.613764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7171.666720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7176.676903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.677111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.680145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.693264] Lustre: Skipped 1 previous similar message [ 7179.306517] Lustre: server umount lustre-OST0000 complete [ 7181.793497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.794513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7181.801223] Lustre: Skipped 1 previous similar message [ 7181.805052] Lustre: Skipped 1 previous similar message [ 7186.913413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7186.917300] Lustre: Skipped 1 previous similar message [ 7187.254402] Lustre: server umount lustre-MDT0000 complete [ 7188.982847] LustreError: 435614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428119 with bad export cookie 16484200231016644882 [ 7188.987343] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7188.990182] LustreError: 435614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7189.149479] Lustre: server umount lustre-MDT0001 complete [ 7193.740172] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7195.188501] Key type lgssc unregistered [ 7195.402786] LNet: 438358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7195.409184] LNetError: 438358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7195.413879] LNet: Removed LNI 192.168.201.114@tcp [ 7196.447204] LNet: Removed LNI 192.168.201.114@tcp100 [ 7196.449795] LNetError: 438358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7196.454574] LNetError: 438358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7196.459714] LNet: Removed LNI 192.168.201.114@tcp101 [ 7197.471202] LNet: Removed LNI 192.168.201.114@tcp102 [ 7197.473781] LNetError: 438358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7197.478465] LNetError: 438358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7197.483602] LNet: Removed LNI 192.168.201.114@tcp103 [ 7198.495183] LNet: Removed LNI 192.168.201.114@tcp104 [ 7199.519195] LNet: Removed LNI 192.168.201.114@tcp105 [ 7199.521823] LNetError: 438358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7199.526612] LNetError: 438358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7199.531761] LNet: Removed LNI 192.168.201.114@tcp106 [ 7200.543171] LNet: Removed LNI 192.168.201.114@tcp107 [ 7200.545833] LNet: Removed LNI 192.168.201.114@tcp108 [ 7200.548559] LNet: Removed LNI 192.168.201.114@tcp109 [ 7200.551224] LNet: Removed LNI 192.168.201.114@tcp110 [ 7200.553814] LNet: Removed LNI 192.168.201.114@tcp111 [ 7200.556457] LNet: Removed LNI 192.168.201.114@tcp112 [ 7200.559142] LNet: Removed LNI 192.168.201.114@tcp113 [ 7200.561719] LNet: Removed LNI 192.168.201.114@tcp114 [ 7200.564366] LNet: Removed LNI 192.168.201.114@tcp115 [ 7200.566905] LNet: Removed LNI 192.168.201.114@tcp116 [ 7200.569536] LNet: Removed LNI 192.168.201.114@tcp117 [ 7200.572158] LNet: Removed LNI 192.168.201.114@tcp118 [ 7200.574685] LNet: Removed LNI 192.168.201.114@tcp119 [ 7200.577332] LNet: Removed LNI 192.168.201.114@tcp120 [ 7200.579857] LNet: Removed LNI 192.168.201.114@tcp121 [ 7200.582564] LNet: Removed LNI 192.168.201.114@tcp122 [ 7200.585182] LNet: Removed LNI 192.168.201.114@tcp123 [ 7200.587693] LNet: Removed LNI 192.168.201.114@tcp124 [ 7200.590348] LNet: Removed LNI 192.168.201.114@tcp125 [ 7200.592883] LNet: Removed LNI 192.168.201.114@tcp126 [ 7200.595517] LNet: Removed LNI 192.168.201.114@tcp127 [ 7200.598148] LNet: Removed LNI 192.168.201.114@tcp128 [ 7200.600702] LNet: Removed LNI 192.168.201.114@tcp129 [ 7200.603349] LNet: Removed LNI 192.168.201.114@tcp130 [ 7200.605878] LNet: Removed LNI 192.168.201.114@tcp131 [ 7200.608496] LNet: Removed LNI 192.168.201.114@tcp132 [ 7200.611143] LNet: Removed LNI 192.168.201.114@tcp133 [ 7200.613662] LNet: Removed LNI 192.168.201.114@tcp134 [ 7200.622651] LNet: Removed LNI 192.168.201.114@tcp135 [ 7201.020981] Key type .llcrypt unregistered [ 7201.023237] Key type ._llcrypt unregistered [ 7211.250384] Key type ._llcrypt registered [ 7211.251854] Key type .llcrypt registered [ 7211.289970] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 7219.230438] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7219.540969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7219.583827] alg: No test for adler32 (adler32-zlib) [ 7220.487912] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 7220.593826] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7222.183171] Key type lgssc registered [ 7222.828482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7240.191249] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:02:49 (1777428169) [ 7248.536338] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7249.943640] Key type lgssc unregistered [ 7250.094684] LNet: 442364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7250.099616] LNetError: 442364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7250.112627] LNet: Removed LNI 192.168.201.114@tcp [ 7250.541761] Key type .llcrypt unregistered [ 7250.543874] Key type ._llcrypt unregistered [ 7258.728926] Key type ._llcrypt registered [ 7258.731109] Key type .llcrypt registered [ 7258.796866] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7259.181934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7259.191832] alg: No test for adler32 (adler32-zlib) [ 7260.057075] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 7260.157688] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7261.759097] Key type lgssc registered [ 7262.258829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7266.787985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7266.794154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.881504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7267.893203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7267.946287] Lustre: lustre-MDT0000: new disk, initializing [ 7267.983851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.995354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7269.795589] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7275.255573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7275.322634] Lustre: 443541:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7275.357435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7275.361316] Lustre: Skipped 1 previous similar message [ 7275.415205] Lustre: lustre-MDT0001: new disk, initializing [ 7275.481731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7275.503810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7275.510159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7276.897979] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7280.856681] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7282.505813] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7285.802358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7285.959278] Lustre: lustre-OST0000: new disk, initializing [ 7285.963184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7285.968610] Lustre: 445283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7286.015925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.580362] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7292.671464] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7293.951179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7293.954738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7293.990787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7294.148143] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing lnet_if_list [ 7296.292559] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7296.301472] LNet: Added LNI 192.168.201.114@tcp10 [8/256/0/180] [ 7298.463152] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7298.482474] LNet: Added LNI 192.168.201.114@tcp11 [8/256/0/180] [ 7300.151873] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7300.170059] LNet: Added LNI 192.168.201.114@tcp12 [8/256/0/180] [ 7302.037389] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7302.057457] LNet: Added LNI 192.168.201.114@tcp13 [8/256/0/180] [ 7304.025368] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7304.045513] LNet: Added LNI 192.168.201.114@tcp14 [8/256/0/180] [ 7305.815736] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7305.835139] LNet: Added LNI 192.168.201.114@tcp15 [8/256/0/180] [ 7307.886909] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7307.905774] LNet: Added LNI 192.168.201.114@tcp16 [8/256/0/180] [ 7309.856201] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7309.877132] LNet: Added LNI 192.168.201.114@tcp17 [8/256/0/180] [ 7311.878353] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7311.897509] LNet: Added LNI 192.168.201.114@tcp18 [8/256/0/180] [ 7313.900448] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7313.918822] LNet: Added LNI 192.168.201.114@tcp19 [8/256/0/180] [ 7316.003204] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7316.021472] LNet: Added LNI 192.168.201.114@tcp20 [8/256/0/180] [ 7318.186623] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7318.198982] LNet: Added LNI 192.168.201.114@tcp21 [8/256/0/180] [ 7320.166149] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7320.184816] LNet: Added LNI 192.168.201.114@tcp22 [8/256/0/180] [ 7322.562362] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7322.581632] LNet: Added LNI 192.168.201.114@tcp23 [8/256/0/180] [ 7324.874075] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7324.891352] LNet: Added LNI 192.168.201.114@tcp24 [8/256/0/180] [ 7327.175324] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7327.194486] LNet: Added LNI 192.168.201.114@tcp25 [8/256/0/180] [ 7329.682245] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7329.700902] LNet: Added LNI 192.168.201.114@tcp26 [8/256/0/180] [ 7332.221903] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7332.234098] LNet: Added LNI 192.168.201.114@tcp27 [8/256/0/180] [ 7334.750124] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7334.769541] LNet: Added LNI 192.168.201.114@tcp28 [8/256/0/180] [ 7337.262133] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7337.280197] LNet: Added LNI 192.168.201.114@tcp29 [8/256/0/180] [ 7339.788215] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7339.807253] LNet: Added LNI 192.168.201.114@tcp30 [8/256/0/180] [ 7342.242334] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7342.261602] LNet: Added LNI 192.168.201.114@tcp31 [8/256/0/180] [ 7344.921061] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7344.930501] LNet: Added LNI 192.168.201.114@tcp32 [8/256/0/180] [ 7347.332928] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7347.342345] LNet: Added LNI 192.168.201.114@tcp33 [8/256/0/180] [ 7349.380678] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7349.400444] LNet: Added LNI 192.168.201.114@tcp34 [8/256/0/180] [ 7352.419384] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7352.437971] LNet: Added LNI 192.168.201.114@tcp35 [8/256/0/180] [ 7354.999551] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7355.012922] LNet: Added LNI 192.168.201.114@tcp36 [8/256/0/180] [ 7357.686736] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7357.707077] LNet: Added LNI 192.168.201.114@tcp37 [8/256/0/180] [ 7361.007287] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7361.026770] LNet: Added LNI 192.168.201.114@tcp38 [8/256/0/180] [ 7364.054093] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7364.063149] LNet: Added LNI 192.168.201.114@tcp39 [8/256/0/180] [ 7365.947125] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7365.955928] LNet: Added LNI 192.168.201.114@tcp40 [8/256/0/180] [ 7367.851536] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7367.860480] LNet: Added LNI 192.168.201.114@tcp41 [8/256/0/180] [ 7369.745861] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7369.754765] LNet: Added LNI 192.168.201.114@tcp42 [8/256/0/180] [ 7371.660628] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7371.669611] LNet: Added LNI 192.168.201.114@tcp43 [8/256/0/180] [ 7373.626580] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7373.635275] LNet: Added LNI 192.168.201.114@tcp44 [8/256/0/180] [ 7375.620832] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7375.629318] LNet: Added LNI 192.168.201.114@tcp45 [8/256/0/180] [ 7377.661539] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7377.670322] LNet: Added LNI 192.168.201.114@tcp46 [8/256/0/180] [ 7379.729131] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7379.738078] LNet: Added LNI 192.168.201.114@tcp47 [8/256/0/180] [ 7381.827109] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7381.835858] LNet: Added LNI 192.168.201.114@tcp48 [8/256/0/180] [ 7383.920529] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7383.929403] LNet: Added LNI 192.168.201.114@tcp49 [8/256/0/180] [ 7386.059873] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7386.068361] LNet: Added LNI 192.168.201.114@tcp50 [8/256/0/180] [ 7393.247680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7393.249442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.252701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7394.272714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7394.273249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7394.274908] Lustre: Skipped 1 previous similar message [ 7394.277682] Lustre: Skipped 1 previous similar message [ 7398.930956] Lustre: server umount lustre-MDT0000 complete [ 7400.064043] LustreError: 445281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428330 with bad export cookie 4194168239239906797 [ 7400.067404] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7400.069297] LustreError: 445281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7400.167029] Lustre: server umount lustre-MDT0001 complete [ 7408.543731] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7412.428443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.871257] Lustre: 442814:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777428334/real 1777428334] req@00000000bfe310c5 x1863768541846912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777428350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7419.886449] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7420.895298] Lustre: 442814:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777428335/real 1777428335] req@0000000076534dd0 x1863768541847040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777428351 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7424.991179] Lustre: 442812:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777428339/real 1777428339] req@0000000000d4c1e1 x1863768541847168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777428355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7426.016495] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3a34adcabc1ece1a [ 7426.021788] Lustre: MGC192.168.201.114@tcp: Connection restored to 0@lo (at 0@lo) [ 7426.172357] LustreError: 445275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.182324] LustreError: 445275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7426.197541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7427.039166] Lustre: 442813:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777428341/real 1777428341] req@00000000649eb568 x1863768541847424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777428357 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7427.760387] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7428.258467] LustreError: 445275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7430.305645] LustreError: 468680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7430.315927] LustreError: 468680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7431.584071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7431.760439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7432.735205] Lustre: 442814:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777428346/real 1777428346] req@0000000089ef9674 x1863768541847552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777428362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7433.451864] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7435.803427] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7436.777309] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7437.468936] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7447.014734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7447.015141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7447.016754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7447.016762] Lustre: Skipped 1 previous similar message [ 7447.019569] LustreError: Skipped 1 previous similar message [ 7447.036283] Lustre: Skipped 1 previous similar message [ 7452.133849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7452.137788] Lustre: Skipped 2 previous similar messages [ 7452.783250] Lustre: server umount lustre-OST0000 complete [ 7457.250767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7457.251233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.264770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7457.268676] Lustre: Skipped 1 previous similar message [ 7460.719320] Lustre: server umount lustre-MDT0000 complete [ 7462.368907] LustreError: 468683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7462.379034] LustreError: 468683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7462.458805] LustreError: 444319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428392 with bad export cookie 4194168239239908890 [ 7462.465154] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7462.466035] LustreError: 444319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7462.616372] Lustre: server umount lustre-MDT0001 complete [ 7467.166631] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7468.617657] Key type lgssc unregistered [ 7468.826959] LNet: 471219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7468.832669] LNetError: 471219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7468.837543] LNet: Removed LNI 192.168.201.114@tcp [ 7468.840058] LNet: Removed LNI 192.168.201.114@tcp10 [ 7468.842668] LNet: Removed LNI 192.168.201.114@tcp11 [ 7468.845228] LNet: Removed LNI 192.168.201.114@tcp12 [ 7468.847822] LNet: Removed LNI 192.168.201.114@tcp13 [ 7468.850312] LNet: Removed LNI 192.168.201.114@tcp14 [ 7468.852848] LNet: Removed LNI 192.168.201.114@tcp15 [ 7468.855510] LNet: Removed LNI 192.168.201.114@tcp16 [ 7468.858059] LNet: Removed LNI 192.168.201.114@tcp17 [ 7468.860680] LNet: Removed LNI 192.168.201.114@tcp18 [ 7468.863174] LNet: Removed LNI 192.168.201.114@tcp19 [ 7468.865798] LNet: Removed LNI 192.168.201.114@tcp20 [ 7468.868331] LNet: Removed LNI 192.168.201.114@tcp21 [ 7468.870829] LNet: Removed LNI 192.168.201.114@tcp22 [ 7468.873471] LNet: Removed LNI 192.168.201.114@tcp23 [ 7468.877539] LNet: Removed LNI 192.168.201.114@tcp24 [ 7468.880052] LNet: Removed LNI 192.168.201.114@tcp25 [ 7468.882641] LNet: Removed LNI 192.168.201.114@tcp26 [ 7468.885189] LNet: Removed LNI 192.168.201.114@tcp27 [ 7468.887747] LNet: Removed LNI 192.168.201.114@tcp28 [ 7468.890266] LNet: Removed LNI 192.168.201.114@tcp29 [ 7468.892794] LNet: Removed LNI 192.168.201.114@tcp30 [ 7468.895186] LNet: Removed LNI 192.168.201.114@tcp31 [ 7468.897681] LNet: Removed LNI 192.168.201.114@tcp32 [ 7468.900099] LNet: Removed LNI 192.168.201.114@tcp33 [ 7468.902559] LNet: Removed LNI 192.168.201.114@tcp34 [ 7468.904927] LNet: Removed LNI 192.168.201.114@tcp35 [ 7468.907237] LNet: Removed LNI 192.168.201.114@tcp36 [ 7468.909479] LNet: Removed LNI 192.168.201.114@tcp37 [ 7468.911732] LNet: Removed LNI 192.168.201.114@tcp38 [ 7468.913965] LNet: Removed LNI 192.168.201.114@tcp39 [ 7468.916228] LNet: Removed LNI 192.168.201.114@tcp40 [ 7468.918259] LNet: Removed LNI 192.168.201.114@tcp41 [ 7468.920258] LNet: Removed LNI 192.168.201.114@tcp42 [ 7468.922233] LNet: Removed LNI 192.168.201.114@tcp43 [ 7468.924169] LNet: Removed LNI 192.168.201.114@tcp44 [ 7468.926088] LNet: Removed LNI 192.168.201.114@tcp45 [ 7468.928035] LNet: Removed LNI 192.168.201.114@tcp46 [ 7468.929945] LNet: Removed LNI 192.168.201.114@tcp47 [ 7468.931818] LNet: Removed LNI 192.168.201.114@tcp48 [ 7468.933648] LNet: Removed LNI 192.168.201.114@tcp49 [ 7468.941463] LNet: Removed LNI 192.168.201.114@tcp50 [ 7469.344064] Key type .llcrypt unregistered [ 7469.346212] Key type ._llcrypt unregistered [ 7478.302880] Key type ._llcrypt registered [ 7478.304496] Key type .llcrypt registered [ 7478.355424] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7478.651081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7478.657849] alg: No test for adler32 (adler32-zlib) [ 7479.518233] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 7479.599101] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7481.191180] Key type lgssc registered [ 7481.745461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7485.263592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7485.275372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7486.469851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.047227] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7492.036125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.211824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7493.999609] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7496.460782] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7497.819178] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7500.459651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7500.623243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.178306] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7504.907610] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7510.885388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7518.496948] loop0: detected capacity change from 0 to 204800000 [ 7518.556995] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7518.559883] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7518.562962] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7520.230147] loop0: detected capacity change from 0 to 204800000 [ 7520.338805] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7520.343946] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7520.349118] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7521.797441] loop0: detected capacity change from 0 to 204800000 [ 7523.411364] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.453441] Lustre: fs15246-OST0000: new disk, initializing [ 7523.455073] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7523.457418] Lustre: 475720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7523.472249] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.055319] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7528.482368] loop1: detected capacity change from 0 to 204800000 [ 7530.099388] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7530.131838] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7530.150410] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7530.184299] Lustre: fs15246-MDT0000: new disk, initializing [ 7530.227235] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7530.234544] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7530.272314] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7531.295706] Lustre: 476823:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000efcf71e8 x1863768771427072/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777428477 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7531.346229] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7536.352981] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7536.355449] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7536.360574] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7540.706544] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.709936] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7544.004645] Lustre: server umount fs15246-MDT0000 complete [ 7545.827045] LustreError: 472436:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7545.831209] LustreError: 472436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7546.596412] Lustre: server umount fs15246-OST0000 complete [ 7550.945314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.946767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7550.948528] Lustre: Skipped 1 previous similar message [ 7550.951017] Lustre: Skipped 1 previous similar message [ 7555.587078] Lustre: server umount lustre-OST0000 complete [ 7558.623751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7558.625448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7558.628365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7561.185168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7561.186138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7562.804448] Lustre: server umount lustre-MDT0000 complete [ 7564.523948] LustreError: 472421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428494 with bad export cookie 1026632344443336279 [ 7564.529909] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7564.531276] LustreError: 472421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7564.685778] Lustre: server umount lustre-MDT0001 complete [ 7569.626184] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7570.454785] Key type lgssc unregistered [ 7570.578302] LNet: 479882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7570.580399] LNetError: 479882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7570.589269] LNet: Removed LNI 192.168.201.114@tcp [ 7570.838423] Key type .llcrypt unregistered [ 7570.839369] Key type ._llcrypt unregistered [ 7577.321205] Key type ._llcrypt registered [ 7577.322190] Key type .llcrypt registered [ 7577.353249] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7577.626972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7577.652049] alg: No test for adler32 (adler32-zlib) [ 7578.491671] Lustre: Lustre: Build Version: 2.17.50_194_g83c1d6d [ 7578.562296] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7580.135086] Key type lgssc registered [ 7580.432571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7591.197458] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 7593.951692] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7604.327722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7604.332668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7605.414333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7605.424227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7605.457547] Lustre: lustre-MDT0000: new disk, initializing [ 7605.479221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7605.484671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7606.589462] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7610.332066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7610.371128] Lustre: 484268:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7610.391126] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7610.392687] Lustre: Skipped 1 previous similar message [ 7610.425724] Lustre: lustre-MDT0001: new disk, initializing [ 7610.455537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7610.466888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7610.469682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7611.568151] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7614.408775] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7615.518243] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7617.517662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7617.610140] Lustre: lustre-OST0000: new disk, initializing [ 7617.611762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7617.614164] Lustre: 486010:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7617.637937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.248599] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7619.511878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7619.514662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7619.550150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7622.114346] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7623.297173] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7623.356521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7624.465269] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7624.523204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7629.794118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7629.797527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7629.800207] LustreError: Skipped 1 previous similar message [ 7629.808922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7629.814569] Lustre: Skipped 1 previous similar message [ 7631.212908] Lustre: server umount lustre-OST0000 complete [ 7634.914547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.915434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7634.921591] Lustre: Skipped 2 previous similar messages [ 7634.928027] Lustre: Skipped 1 previous similar message [ 7639.156757] Lustre: server umount lustre-MDT0000 complete [ 7640.743105] LustreError: 484258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777428571 with bad export cookie 1800531593982363832 [ 7640.747388] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7640.750436] LustreError: 484258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7640.905288] Lustre: server umount lustre-MDT0001 complete [ 7650.991439] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 7653.847095] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7663.411845] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7477 sec ========== 22:09:53 (1777428593) [ 7663.932500] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:09:53 (1777428593) === [ 7665.007406] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:09:54 (1777428594) === [ 7676.445730] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7677.290162] Key type lgssc unregistered [ 7677.400275] LNet: 492443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7677.402272] LNetError: 492443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7677.409301] LNet: Removed LNI 192.168.201.114@tcp [ 7677.655186] Key type .llcrypt unregistered [ 7677.656132] Key type ._llcrypt unregistered