[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 517039750 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2560MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003269] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008695] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011015] pid_max: default: 32768 minimum: 301 [ 0.012153] LSM: Security Framework initializing [ 0.013061] Yama: becoming mindful. [ 0.014051] SELinux: Initializing. [ 0.015079] *** VALIDATE selinux *** [ 0.023611] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028572] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031146] *** VALIDATE tmpfs *** [ 0.033317] *** VALIDATE proc *** [ 0.034374] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.036313] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037177] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039042] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043124] debug: unmapping init [mem 0xffffffff9de59000-0xffffffff9de60fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046770] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047034] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051020] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054356] rcu: Hierarchical SRCU implementation. [ 0.056579] smp: Bringing up secondary CPUs ... [ 0.057581] x86: Booting SMP configuration: [ 0.058039] .... node #0, CPUs: #1 #2 #3 [ 0.061425] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148506] node 0 deferred pages initialised in 82ms [ 0.152106] devtmpfs: initialized [ 0.153268] x86/mm: Memory block size: 128MB [ 0.156993] gcov: version magic: 0x41383552 [ 0.160353] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168327] pinctrl core: initialized pinctrl subsystem [ 0.170199] [ 0.170889] ************************************************************* [ 0.174015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177009] ** ** [ 0.180013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.183013] ** ** [ 0.185011] ** This means that this kernel is built to expose internal ** [ 0.188015] ** IOMMU data structures, which may compromise security on ** [ 0.191017] ** your system. ** [ 0.194015] ** ** [ 0.197014] ** If you see this message and you are not debugging the ** [ 0.200016] ** kernel, report this immediately to your vendor! ** [ 0.203013] ** ** [ 0.206014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209028] ************************************************************* [ 0.212694] NET: Registered protocol family 16 [ 0.215512] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.218064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.222073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.226105] cpuidle: using governor menu [ 0.227924] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228640] PCI: Using configuration type 1 for base access [ 0.229155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.237140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.241069] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.245157] cryptd: max_cpu_qlen set to 1000 [ 0.250430] ACPI: Added _OSI(Module Device) [ 0.251029] ACPI: Added _OSI(Processor Device) [ 0.252023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.253018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.257502] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.264817] ACPI: Interpreter enabled [ 0.266151] ACPI: PM: (supports S0 S3 S4 S5) [ 0.268020] ACPI: Using IOAPIC for interrupt routing [ 0.270153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.274486] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.285567] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.288038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.291024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.295081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.301489] acpiphp: Slot [2] registered [ 0.303167] acpiphp: Slot [5] registered [ 0.305196] acpiphp: Slot [6] registered [ 0.307139] acpiphp: Slot [7] registered [ 0.308131] acpiphp: Slot [8] registered [ 0.310168] acpiphp: Slot [9] registered [ 0.312138] acpiphp: Slot [10] registered [ 0.314157] acpiphp: Slot [3] registered [ 0.316128] acpiphp: Slot [4] registered [ 0.318131] acpiphp: Slot [11] registered [ 0.320124] acpiphp: Slot [12] registered [ 0.322136] acpiphp: Slot [13] registered [ 0.324264] acpiphp: Slot [14] registered [ 0.326119] acpiphp: Slot [15] registered [ 0.328122] acpiphp: Slot [16] registered [ 0.330119] acpiphp: Slot [17] registered [ 0.332208] acpiphp: Slot [18] registered [ 0.333208] acpiphp: Slot [19] registered [ 0.336238] acpiphp: Slot [20] registered [ 0.337191] acpiphp: Slot [21] registered [ 0.339195] acpiphp: Slot [22] registered [ 0.341147] acpiphp: Slot [23] registered [ 0.343171] acpiphp: Slot [24] registered [ 0.345104] acpiphp: Slot [25] registered [ 0.347139] acpiphp: Slot [26] registered [ 0.348114] acpiphp: Slot [27] registered [ 0.350133] acpiphp: Slot [28] registered [ 0.352125] acpiphp: Slot [29] registered [ 0.353124] acpiphp: Slot [30] registered [ 0.355145] acpiphp: Slot [31] registered [ 0.357071] PCI host bridge to bus 0000:00 [ 0.359025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.361032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.364031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.367027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.370032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.374095] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.376270] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.380682] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.385895] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.397977] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.405225] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.409036] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.412032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.415019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.418491] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.421918] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.425050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.428859] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.435017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.448014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.453013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.459836] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.472018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.480017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.500017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.516249] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.529018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.543025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.569019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.590470] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.604017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.613019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.637018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.652987] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.666017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.675017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.696071] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.708850] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.717017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.729013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.746014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.760061] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.774018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.786018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.804020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.826380] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.829372] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.831406] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.833353] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.836229] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.841100] iommu: Default domain type: Passthrough [ 0.843609] SCSI subsystem initialized [ 0.845267] ACPI: bus type USB registered [ 0.847128] usbcore: registered new interface driver usbfs [ 0.850107] usbcore: registered new interface driver hub [ 0.852083] usbcore: registered new device driver usb [ 0.854218] pps_core: LinuxPPS API ver. 1 registered [ 0.857015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.861137] PTP clock support registered [ 0.863121] EDAC MC: Ver: 3.0.0 [ 0.866145] PCI: Using ACPI for IRQ routing [ 0.867926] NetLabel: Initializing [ 0.868011] NetLabel: domain hash size = 128 [ 0.869011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.870118] NetLabel: unlabeled traffic allowed by default [ 0.872147] vgaarb: loaded [ 0.875229] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.877017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.884713] clocksource: Switched to clocksource kvm-clock [ 1.010199] VFS: Disk quotas dquot_6.6.0 [ 1.012197] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.014074] *** VALIDATE ramfs *** [ 1.014934] *** VALIDATE hugetlbfs *** [ 1.015948] pnp: PnP ACPI init [ 1.017812] pnp: PnP ACPI: found 6 devices [ 1.040724] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.044384] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.046783] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.049346] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.051993] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.054642] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.057743] NET: Registered protocol family 2 [ 1.060332] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.065814] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.069776] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.075661] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.079284] TCP: Hash tables configured (established 65536 bind 65536) [ 1.082548] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.085833] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.089543] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.092449] NET: Registered protocol family 1 [ 1.094804] RPC: Registered named UNIX socket transport module. [ 1.096969] RPC: Registered udp transport module. [ 1.098560] RPC: Registered tcp transport module. [ 1.100301] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.102529] NET: Registered protocol family 44 [ 1.104077] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.106212] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.108182] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.110416] PCI: CLS 0 bytes, default 64 [ 1.112733] Unpacking initramfs... [ 2.534094] debug: unmapping init [mem 0xffff8ff77cc54000-0xffff8ff77ffbffff] [ 2.538127] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.540450] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.543436] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.096401] Initialise system trusted keyrings [ 3.098197] Key type blacklist registered [ 3.100068] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.109138] zbud: loaded [ 3.111939] *** VALIDATE nfs *** [ 3.113177] *** VALIDATE nfs4 *** [ 3.114703] pstore: using deflate compression [ 3.117972] Platform Keyring initialized [ 3.219989] NET: Registered protocol family 38 [ 3.221801] Key type asymmetric registered [ 3.223824] Asymmetric key parser 'x509' registered [ 3.226156] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.229397] io scheduler mq-deadline registered [ 3.232927] io scheduler kyber registered [ 3.234995] io scheduler bfq registered [ 3.237638] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.241460] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.244462] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.247629] ACPI: Power Button [PWRF] [ 3.253807] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.261043] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.275895] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.283261] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.298489] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.326232] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.356043] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.361416] Non-volatile memory driver v1.3 [ 3.363196] Linux agpgart interface v0.103 [ 3.398319] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.401511] vda: detected capacity change from 0 to 75071488 [ 3.423648] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.427419] vdb: detected capacity change from 0 to 1073741824 [ 3.444308] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.448029] vdc: detected capacity change from 0 to 2621440000 [ 3.464343] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.467514] vdd: detected capacity change from 0 to 2621440000 [ 3.482296] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.485469] vde: detected capacity change from 0 to 4294967296 [ 3.503240] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.506549] vdf: detected capacity change from 0 to 4294967296 [ 3.521980] libphy: Fixed MDIO Bus: probed [ 3.531967] usbcore: registered new interface driver usbserial_generic [ 3.535190] usbserial: USB Serial support registered for generic [ 3.537906] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.542262] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.544213] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.546717] mousedev: PS/2 mouse device common for all mice [ 3.550272] rtc_cmos 00:05: RTC can wake from S4 [ 3.553170] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.557164] rtc_cmos 00:05: registered as rtc0 [ 3.560138] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.563100] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.564225] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.570958] intel_pstate: CPU model not supported [ 3.574178] hid: raw HID events driver (C) Jiri Kosina [ 3.576374] usbcore: registered new interface driver usbhid [ 3.578484] usbhid: USB HID core driver [ 3.580290] drop_monitor: Initializing network drop monitor service [ 3.582767] Initializing XFRM netlink socket [ 3.584951] NET: Registered protocol family 10 [ 3.588548] Segment Routing with IPv6 [ 3.590178] NET: Registered protocol family 17 [ 3.592408] mpls_gso: MPLS GSO support [ 3.597701] RAS: Correctable Errors collector initialized. [ 3.599983] AVX version of gcm_enc/dec engaged. [ 3.601724] AES CTR mode by8 optimization enabled [ 3.676657] sched_clock: Marking stable (3676636538, 0)->(4646479393, -969842855) [ 3.680588] registered taskstats version 1 [ 3.682680] Loading compiled-in X.509 certificates [ 3.685363] zswap: loaded using pool lzo/zbud [ 3.707202] Key type big_key registered [ 3.721084] Key type encrypted registered [ 3.722890] ima: No TPM chip found, activating TPM-bypass! [ 3.724905] ima: Allocated hash algorithm: sha1 [ 3.726321] ima: No architecture policies found [ 3.728483] evm: Initialising EVM extended attributes: [ 3.730989] evm: security.selinux [ 3.732566] evm: security.ima [ 3.734111] evm: security.capability [ 3.735493] evm: HMAC attrs: 0x1 [ 3.738167] rtc_cmos 00:05: setting system clock to 2026-08-26 22:19:03 UTC (1787782743) [ 3.744676] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 3.747908] debug: unmapping init [mem 0xffffffff9db82000-0xffffffff9de58fff] [ 3.755131] Write protecting the kernel read-only data: 28672k [ 3.758443] debug: unmapping init [mem 0xffffffff9c203000-0xffffffff9c3fffff] [ 3.761219] debug: unmapping init [mem 0xffffffff9cb14000-0xffffffff9cbfffff] [ 3.794862] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.803358] systemd[1]: Detected virtualization kvm. [ 3.805255] systemd[1]: Detected architecture x86-64. [ 3.806782] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.834548] systemd[1]: No hostname configured. [ 3.836198] systemd[1]: Set hostname to . [ 3.838220] random: systemd: uninitialized urandom read (16 bytes read) [ 3.840627] systemd[1]: Initializing machine ID from random generator. [ 3.966951] random: systemd: uninitialized urandom read (16 bytes read) [ 3.969179] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.972376] random: systemd: uninitialized urandom read (16 bytes read) [ 3.973778] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.976186] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.621929] device-mapper: uevent: version 1.0.3 [ 4.623975] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.396404] random: fast init done [ 5.464756] scsi host0: ata_piix [ 5.628450] scsi host1: ata_piix [ 5.631628] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.634745] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.655449] virtio_net virtio0 ens2: renamed from eth0 [ 9.512890] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.241555] random: crng init done [ 10.242914] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.786312] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.740876] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.776879] SELinux: Disabled at runtime. [ 18.079295] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.129826] systemd[1]: Detected virtualization kvm. [ 18.151831] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.152872] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.167685] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.183716] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.193258] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.209856] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.250039] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.286029] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ 20.425824] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 22.634735] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.861248] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.948274] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.650670] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.812827] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 31.607808] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 33.024945] NFS: Registering the id_resolver key type [ 33.034447] Key type id_resolver registered [ 33.037698] Key type id_legacy registered [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg114-server login: [ 93.870363] libcfs: loading out-of-tree module taints kernel. [ 93.924113] Key type ._llcrypt registered [ 93.925982] Key type .llcrypt registered [ 94.065440] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 117.279853] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 119.948769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 119.973176] alg: No test for adler32 (adler32-zlib) [ 121.616654] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 122.590668] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 124.455177] Key type lgssc registered [ 127.144091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.337327] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.323302] hrtimer: interrupt took 2487469 ns [ 187.933512] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 200.987776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.019554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.416928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.444604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.560915] Lustre: lustre-MDT0000: new disk, initializing [ 202.689876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.708933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.695672] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.798253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.967711] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 221.016152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 221.019851] Lustre: Skipped 1 previous similar message [ 221.198796] Lustre: lustre-MDT0001: new disk, initializing [ 221.300302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.342219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 221.353140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.555715] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.347925] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.882914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.343389] Lustre: lustre-OST0000: new disk, initializing [ 245.347102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.351386] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.433660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.839916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.846323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 246.959806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.078855] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.950760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.137210] Lustre: lustre-OST0001: new disk, initializing [ 267.148482] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 267.151742] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.263793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.155775] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.542302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.558893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.646921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 286.576861] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.392655] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 302.307738] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing check_logdir /tmp/testlogs/ [ 308.693511] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing yml_node [ 313.786822] Lustre: DEBUG MARKER: Client: 2.17.57.55 [ 316.449987] Lustre: DEBUG MARKER: MDS: 2.17.57.55 [ 319.446257] Lustre: DEBUG MARKER: OSS: 2.17.57.55 [ 321.149584] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 18:24:18 EDT 2026 [ 338.260482] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 339.636305] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 357.874937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.895909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 357.907580] Lustre: Skipped 3 previous similar messages [ 357.918413] Lustre: Skipped 3 previous similar messages [ 361.534249] Lustre: server umount lustre-MDT0000 complete [ 365.781249] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783105 with bad export cookie 4322315026649874094 [ 365.781450] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 365.798717] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 368.096535] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.099810] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.100179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.133566] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 368.141621] Lustre: Skipped 1 previous similar message [ 370.155678] LustreError: 8540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.159444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.187501] LustreError: 8540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 370.194409] Lustre: Skipped 2 previous similar messages [ 375.267382] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.267689] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 375.284299] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 375.309128] Lustre: Skipped 1 previous similar message [ 380.384602] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 380.397368] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.397455] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.436426] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 380.737434] Lustre: server umount lustre-MDT0001 complete [ 385.307648] Lustre: server umount lustre-OST0000 complete [ 389.073861] Lustre: server umount lustre-OST0001 complete [ 396.755219] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 406.817183] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 445.518801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.791494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 445.845321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 445.965929] Lustre: lustre-MDT0000: new disk, initializing [ 446.049709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.096294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 451.277954] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 462.628609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.771632] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.859666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.863236] Lustre: Skipped 1 previous similar message [ 462.957191] Lustre: lustre-MDT0001: new disk, initializing [ 463.121816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.169366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.181749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.451348] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 478.406559] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 484.778157] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 491.237152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.693721] Lustre: lustre-OST0000: new disk, initializing [ 491.697616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 491.703210] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 491.832235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.649518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 493.668286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 493.804095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 498.179458] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 510.181854] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 516.812543] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 517.191680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 524.114715] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 524.527265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 529.895995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 529.897657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.911502] LustreError: Skipped 1 previous similar message [ 529.925321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.935098] Lustre: Skipped 1 previous similar message [ 529.960891] Lustre: Skipped 2 previous similar messages [ 533.799084] Lustre: server umount lustre-OST0000 complete [ 540.128935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.136680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.151171] Lustre: Skipped 1 previous similar message [ 544.066168] Lustre: server umount lustre-MDT0000 complete [ 545.264235] LustreError: 15967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.293894] LustreError: 15967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 548.893591] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783288 with bad export cookie 4322315026649875627 [ 548.895430] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.910147] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 549.381742] Lustre: server umount lustre-MDT0001 complete [ 555.715247] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:28:12 (1787783292) [ 558.621587] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 560.916712] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:28:18 (1787783298) [ 569.429990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.982779] LustreError: 19466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.163384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.164885] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 586.552417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.286414] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 600.951831] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 606.896624] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 614.730519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 614.826960] LustreError: 21287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.846945] LustreError: 21287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 622.645592] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 632.885650] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 642.024083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.025830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 642.035042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 642.051981] Lustre: Skipped 2 previous similar messages [ 647.831643] Lustre: server umount lustre-OST0000 complete [ 652.256401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.272802] Lustre: Skipped 1 previous similar message [ 657.800291] Lustre: server umount lustre-MDT0000 complete [ 662.360234] LustreError: 21309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783402 with bad export cookie 4322315026649876509 [ 662.368589] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.380677] LustreError: 21309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 662.932806] Lustre: server umount lustre-MDT0001 complete [ 676.608552] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 679.310936] Key type lgssc unregistered [ 679.644222] LNet: 23002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.650749] LNetError: 23002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.689361] LNet: Removed LNI 192.168.201.114@tcp [ 680.704990] Key type .llcrypt unregistered [ 680.706738] Key type ._llcrypt unregistered [ 705.394275] Key type ._llcrypt registered [ 705.396594] Key type .llcrypt registered [ 705.500621] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 706.821853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 706.852449] alg: No test for adler32 (adler32-zlib) [ 707.984645] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 708.300845] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 709.967849] Key type lgssc registered [ 711.560693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 720.634948] loop: module loaded [ 720.753612] loop0: detected capacity change from 0 to 204800000 [ 720.976260] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 720.989051] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.315665] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 727.505699] loop0: detected capacity change from 0 to 204800000 [ 727.838609] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 727.849968] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 727.897778] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.086616] loop0: detected capacity change from 0 to 204800000 [ 738.715722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 738.771806] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.251854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 740.374312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 740.513782] Lustre: lustre-MDT0000: new disk, initializing [ 740.718510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 740.763069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 746.151075] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 756.078095] loop1: detected capacity change from 0 to 204800000 [ 762.657290] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 763.070276] Lustre: lustre-OST07c6: new disk, initializing [ 763.077958] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 763.086773] Lustre: Skipped 1 previous similar message [ 763.106088] Lustre: 26192:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 763.232330] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 769.720635] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 772.225533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 772.232813] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 772.265740] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 774.400210] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 774.410099] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 774.416286] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 774.427227] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 774.435142] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 774.444296] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 792.809314] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 792.819791] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 792.826591] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 792.846876] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 792.852428] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 792.857210] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 798.026846] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 798.040876] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 798.049601] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 798.055794] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.062583] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 798.068687] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.074363] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 798.082177] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.087584] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 798.094876] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.101098] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 798.108604] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.278678] Lustre: 26199:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 803.289651] Lustre: 26199:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 803.295452] Lustre: 26199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 803.300344] Lustre: 26199:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.307384] Lustre: 26199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 803.320762] Lustre: 26199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.332426] Lustre: 26199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 803.344908] Lustre: 26199:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.350722] Lustre: 26199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 803.359965] Lustre: 26199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.367281] Lustre: 26199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 803.373216] Lustre: 26199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.463212] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 811.472421] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 811.478198] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 811.483385] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.494619] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 811.504886] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.513929] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 811.526292] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.540537] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 811.549565] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.561034] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 811.572437] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 819.518241] Lustre: 26199:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 819.531820] Lustre: 26199:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 819.542141] Lustre: 26199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 819.552921] Lustre: 26199:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 819.562672] Lustre: 26199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 819.568984] Lustre: 26199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 819.574216] Lustre: 26199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 819.580290] Lustre: 26199:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 819.590264] Lustre: 26199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 819.607782] Lustre: 26199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 819.624938] Lustre: 26199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 819.633286] Lustre: 26199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 869.345129] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 869.350871] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 869.404783] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 874.467885] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 879.598097] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 882.145053] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 882.426235] Lustre: server umount lustre-OST07c6 complete [ 892.592146] Lustre: server umount lustre-MDT0000 complete [ 912.677530] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 916.198418] Key type lgssc unregistered [ 916.533973] LNet: 28753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.542430] LNetError: 28753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.627871] LNet: Removed LNI 192.168.201.114@tcp [ 918.589554] Key type .llcrypt unregistered [ 918.595884] Key type ._llcrypt unregistered [ 938.170432] Key type ._llcrypt registered [ 938.175770] Key type .llcrypt registered [ 938.305070] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 939.728931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 939.749725] alg: No test for adler32 (adler32-zlib) [ 941.014639] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 941.422023] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 943.135346] Key type lgssc registered [ 944.599301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 953.097452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 953.147786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.899582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.006497] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 965.096203] LustreError: 30025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.165573] LustreError: 30025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 970.210931] LustreError: 30026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.232375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.985335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 979.708762] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 988.264530] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 995.145265] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1003.290356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1003.776929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.968370] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1021.545314] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1025.271730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1044.966819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1044.984481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.002906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1045.651682] Lustre: server umount lustre-OST0000 complete [ 1051.113712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.123503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.127301] Lustre: Skipped 1 previous similar message [ 1051.142855] Lustre: Skipped 1 previous similar message [ 1056.224889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.235763] Lustre: Skipped 1 previous similar message [ 1060.329876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1060.334708] Lustre: Skipped 1 previous similar message [ 1060.549441] Lustre: server umount lustre-MDT0000 complete [ 1064.817456] LustreError: 30007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783804 with bad export cookie 7120741575771177308 [ 1064.831106] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1065.136251] Lustre: server umount lustre-MDT0001 complete [ 1078.648878] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1082.179309] Key type lgssc unregistered [ 1082.513586] LNet: 33765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.521394] LNetError: 33765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.613501] LNet: Removed LNI 192.168.201.114@tcp [ 1084.646192] Key type .llcrypt unregistered [ 1084.648927] Key type ._llcrypt unregistered [ 1105.401963] Key type ._llcrypt registered [ 1105.404247] Key type .llcrypt registered [ 1105.568656] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1106.809567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1106.919085] alg: No test for adler32 (adler32-zlib) [ 1107.995988] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1108.253660] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1109.975226] Key type lgssc registered [ 1111.187592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1121.099507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1121.118730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.730598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.155483] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1133.043855] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.077403] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1138.157334] LustreError: 35041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.688058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.418964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1146.405656] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1155.078895] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1162.033121] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1169.409915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1169.676643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.961668] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1185.997579] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1193.300503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1208.838225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.850891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1208.873516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.894989] Lustre: Skipped 1 previous similar message [ 1210.416539] Lustre: server umount lustre-OST0000 complete [ 1216.994338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1217.009124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.038397] Lustre: Skipped 1 previous similar message [ 1217.048348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.041743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.053152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.073471] Lustre: Skipped 1 previous similar message [ 1222.428630] Lustre: server umount lustre-MDT0000 complete [ 1224.160878] LustreError: 35035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1224.172534] LustreError: 35035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1226.841197] LustreError: 36886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783966 with bad export cookie 4771078956595754762 [ 1226.853829] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1226.861983] LustreError: 36886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1227.317920] Lustre: server umount lustre-MDT0001 complete [ 1241.573795] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1245.234757] Key type lgssc unregistered [ 1245.755826] LNet: 38600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.771597] LNetError: 38600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.828611] LNet: Removed LNI 192.168.201.114@tcp [ 1248.085175] Key type .llcrypt unregistered [ 1248.087675] Key type ._llcrypt unregistered [ 1279.306616] Key type ._llcrypt registered [ 1279.309232] Key type .llcrypt registered [ 1279.422076] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1280.825418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.916384] alg: No test for adler32 (adler32-zlib) [ 1282.094568] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1282.340068] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1284.055649] Key type lgssc registered [ 1285.422621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1294.096124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1294.118472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.782520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1301.509974] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1306.106168] LustreError: 40018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.132524] LustreError: 40018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1311.201946] LustreError: 40019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.404392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.990643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1317.966875] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1326.796842] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1332.766779] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1340.661064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1340.856703] LustreError: 41841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.013845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.057453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1349.380354] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1358.484942] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1365.887510] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 1366.513092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.532414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.548973] Lustre: Skipped 2 previous similar messages [ 1366.569463] Lustre: Skipped 2 previous similar messages [ 1370.986419] LustreError: 40013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.213302] Lustre: server umount lustre-MDT0000 complete [ 1375.969056] LustreError: 39999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784115 with bad export cookie 1400200287549542557 [ 1375.971219] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.980412] LustreError: 39999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1376.093249] LustreError: 40014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.098380] LustreError: 40014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1376.127401] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 1376.740940] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.796047] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1380.810232] Lustre: Skipped 1 previous similar message [ 1383.016692] Lustre: server umount lustre-MDT0001 complete [ 1405.889728] Lustre: server umount lustre-OST0000 complete [ 1426.736536] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1430.054964] Key type lgssc unregistered [ 1430.362392] LNet: 43883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.369161] LNetError: 43883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.409512] LNet: Removed LNI 192.168.201.114@tcp [ 1432.340740] Key type .llcrypt unregistered [ 1432.342818] Key type ._llcrypt unregistered [ 1463.120469] Key type ._llcrypt registered [ 1463.124489] Key type .llcrypt registered [ 1463.290566] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1464.364466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.512188] alg: No test for adler32 (adler32-zlib) [ 1465.921759] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1466.272310] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1467.959631] Key type lgssc registered [ 1469.205340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.226851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1477.252617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.769386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.331940] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1488.358582] LustreError: 45302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.372894] LustreError: 45302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1493.484778] LustreError: 45303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1493.693268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.190379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1499.407888] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1509.560571] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1516.206433] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1525.294256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1525.803237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.806842] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1542.831057] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1549.435587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1559.521424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1559.528432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.561397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1560.041168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.062550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1560.067677] Lustre: Skipped 1 previous similar message [ 1564.066594] Lustre: lustre-OST0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 1565.570826] Lustre: server umount lustre-OST0000 complete [ 1585.648707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.654444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1585.675978] Lustre: Skipped 1 previous similar message [ 1585.686384] Lustre: Skipped 3 previous similar messages [ 1590.755278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.762470] Lustre: Skipped 1 previous similar message [ 1591.197661] Lustre: server umount lustre-MDT0000 complete [ 1595.874078] LustreError: 45302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.892396] LustreError: 45302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1596.620957] LustreError: 45282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784336 with bad export cookie 6400659721051621282 [ 1596.633365] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1596.638165] LustreError: 45282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1603.371497] Lustre: server umount lustre-MDT0001 complete [ 1617.343401] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1621.178818] Key type lgssc unregistered [ 1621.535171] LNet: 49038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.550241] LNetError: 49038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.576596] LNet: Removed LNI 192.168.201.114@tcp [ 1622.492903] Key type .llcrypt unregistered [ 1622.498318] Key type ._llcrypt unregistered [ 1650.374262] Key type ._llcrypt registered [ 1650.375765] Key type .llcrypt registered [ 1650.494904] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1651.884731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1652.097652] alg: No test for adler32 (adler32-zlib) [ 1653.360366] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1653.724112] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1655.519342] Key type lgssc registered [ 1656.742076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1666.312145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1666.343936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1668.015187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.391631] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1677.286408] LustreError: 50463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.309882] LustreError: 50463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1682.406114] LustreError: 50464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1683.763257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.418540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.822954] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1699.026368] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1706.205332] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1714.795419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1715.395878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.304677] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1730.834771] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1737.906498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1746.722419] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1750.378245] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1764.066573] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2618h48m19s [ 1766.054194] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1768.945352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.953169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1768.984696] Lustre: Skipped 2 previous similar messages [ 1768.990858] Lustre: Skipped 1 previous similar message [ 1770.029135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.051869] Lustre: Skipped 2 previous similar messages [ 1773.419353] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 1774.835755] Lustre: server umount lustre-MDT0000 complete [ 1776.104812] LustreError: 50460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1776.123305] LustreError: 50460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1779.232909] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1780.578841] LustreError: 51560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1786.736030] LustreError: 50460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1786.757180] LustreError: 50460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1790.367709] Lustre: 49775:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787784514/real 1787784514] req@0000000083f4b0ab x1874627807719040/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787784530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1790.438531] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.683469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.512359] LustreError: 51191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.527834] LustreError: 51191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 1800.617326] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf32f4e4d1355ed3a [ 1800.627415] Lustre: MGC192.168.201.114@tcp: Connection restored to 0@lo (at 0@lo) [ 1800.884321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.930978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1801.902463] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1806.362241] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1810.818601] Lustre: DEBUG MARKER: Wait for df (36741) ... [ 1813.187922] Lustre: DEBUG MARKER: done [ 1817.575755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.594228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.596262] Lustre: Skipped 1 previous similar message [ 1822.177197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1822.182089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.193453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.800379] Lustre: server umount lustre-OST0000 complete [ 1827.811090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.813265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1827.826398] Lustre: Skipped 1 previous similar message [ 1833.484516] Lustre: server umount lustre-MDT0000 complete [ 1833.963768] LustreError: 50463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.000104] LustreError: 50463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1837.881669] LustreError: 50443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784577 with bad export cookie 17523310768065735994 [ 1837.890820] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1837.893274] LustreError: 50443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1838.613620] Lustre: server umount lustre-MDT0001 complete [ 1852.854101] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1856.231871] Key type lgssc unregistered [ 1856.604495] LNet: 55416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.613475] LNetError: 55416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.633402] LNet: Removed LNI 192.168.201.114@tcp [ 1857.865388] Key type .llcrypt unregistered [ 1857.866849] Key type ._llcrypt unregistered [ 1904.495614] Key type ._llcrypt registered [ 1904.499636] Key type .llcrypt registered [ 1904.563534] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1905.404861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1905.443500] alg: No test for adler32 (adler32-zlib) [ 1906.562362] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1906.828900] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1908.495968] Key type lgssc registered [ 1909.417299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1919.154060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1919.182662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.506604] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1920.558369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1920.838266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.514987] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1938.466737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.569822] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1938.603239] Lustre: 57629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1939.048379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1944.321540] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1952.633591] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1959.062961] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1966.772446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.052774] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1967.364123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.863314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1973.584566] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1981.467247] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1988.534204] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2618h52m05s [ 1990.345607] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2001.478360] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2005.478091] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2005.552557] LustreError: 57636:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2025.852910] Lustre: lustre-MDT0000: Client 10d6f695-8804-466a-b068-0e6f662a3112 (at 192.168.201.14@tcp) reconnecting [ 2045.607140] LustreError: 57636:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2048.438898] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2051.095348] Lustre: DEBUG MARKER: done [ 2059.752482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2059.762503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.786403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2064.158720] Lustre: server umount lustre-OST0000 complete [ 2070.506997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.524669] Lustre: Skipped 1 previous similar message [ 2070.548817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2070.560363] Lustre: Skipped 3 previous similar messages [ 2071.592751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.602701] Lustre: Skipped 1 previous similar message [ 2077.666026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.672298] Lustre: Skipped 1 previous similar message [ 2082.271740] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2082.699146] Lustre: server umount lustre-MDT0000 complete [ 2086.901904] LustreError: 57621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784826 with bad export cookie 10690401301713952771 [ 2086.909927] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2086.922975] LustreError: 57621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2087.168975] Lustre: server umount lustre-MDT0001 complete [ 2100.989828] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2104.332285] Key type lgssc unregistered [ 2104.736284] LNet: 61964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.756822] LNetError: 61964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.780061] LNet: Removed LNI 192.168.201.114@tcp [ 2105.892645] Key type .llcrypt unregistered [ 2105.894656] Key type ._llcrypt unregistered [ 2156.330748] Key type ._llcrypt registered [ 2156.334762] Key type .llcrypt registered [ 2156.440655] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2158.089384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2158.130136] alg: No test for adler32 (adler32-zlib) [ 2159.338929] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2159.759370] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2161.519692] Key type lgssc registered [ 2162.929220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2166.853836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2166.873837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.356304] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2168.375529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2168.519224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.660350] Lustre: Failing over lustre-MDT0000 [ 2169.855697] Lustre: server umount lustre-MDT0000 complete [ 2177.470508] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:55:14 (1787784914) [ 2185.443681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.754421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.793369] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2200.506842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.647741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2200.689286] Lustre: 64764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2200.965285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2206.394873] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2214.713387] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2221.015458] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2228.591265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2228.877500] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2229.231539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.649646] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2237.742344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2245.125270] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2252.075082] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2262.500936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2262.509265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2262.520687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.522177] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.533568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.546874] Lustre: Skipped 1 previous similar message [ 2263.574113] Lustre: Skipped 2 previous similar messages [ 2268.642760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.655150] Lustre: Skipped 2 previous similar messages [ 2273.247152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2273.480946] Lustre: server umount lustre-MDT0000 complete [ 2277.565658] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785017 with bad export cookie 18263935789125362293 [ 2277.590391] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.962826] Lustre: server umount lustre-MDT0001 complete [ 2282.257270] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2287.179407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2294.239542] Lustre: 63686:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787785018/real 1787785018] req@00000000a8a7dc79 x1874628338298368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787785034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2294.279120] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.120479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.340291] Lustre: 63688:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787785022/real 1787785022] req@000000005188c9d4 x1874628338298496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787785038 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2299.360074] Lustre: 63689:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787785023/real 1787785023] req@00000000917f8924 x1874628338298624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787785039 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2302.935736] LustreError: 66613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.965928] LustreError: 66613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2303.129445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.276273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2304.033410] Lustre: 63686:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787785027/real 1787785027] req@00000000dcb39d94 x1874628338298880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787785043 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2304.167162] LustreError: 66614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.179234] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2304.202394] LustreError: 66614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2305.253335] LustreError: 68541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.273215] LustreError: 68541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2308.528343] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2309.087771] Lustre: 63686:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787785032/real 1787785032] req@00000000446560b8 x1874628338299008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787785048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2310.630608] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.653949] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2315.749455] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.780570] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2317.946943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.442987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2323.234332] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2323.938704] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2332.243368] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2341.584617] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2354.151436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2354.165481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.177126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2357.528097] Lustre: server umount lustre-MDT0000 complete [ 2359.791283] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.814046] LustreError: 68540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2361.115265] LustreError: 64755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785100 with bad export cookie 18263935789125364568 [ 2361.118263] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2361.121495] LustreError: 64755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2361.437464] Lustre: server umount lustre-MDT0001 complete [ 2368.776910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2379.563605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.255114] Lustre: 63689:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787785104/real 1787785104] req@00000000e6d4ca8a x1874628338328320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787785120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2380.286262] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.306148] Lustre: Skipped 2 previous similar messages [ 2385.882350] LustreError: 66614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.033520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.132611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2391.235499] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2401.309960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.889283] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2407.643456] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2410.030296] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2416.759757] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2424.236174] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2432.481031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2432.495605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.525059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2432.537853] Lustre: Skipped 2 previous similar messages [ 2438.796407] Lustre: server umount lustre-MDT0000 complete [ 2440.675804] LustreError: 66613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.701120] LustreError: 66613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2442.937332] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785182 with bad export cookie 18263935789125365947 [ 2442.942715] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2442.960201] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2443.390654] Lustre: server umount lustre-MDT0001 complete [ 2450.975198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2455.272099] Lustre: DEBUG MARKER: files compared the same [ 2468.310056] Lustre: server umount lustre-OST0000 complete [ 2488.651700] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2492.053932] Key type lgssc unregistered [ 2492.405585] LNet: 74505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2492.419902] LNetError: 74505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2492.438945] LNet: Removed LNI 192.168.201.114@tcp [ 2493.222139] Key type .llcrypt unregistered [ 2493.224160] Key type ._llcrypt unregistered [ 2520.934776] Key type ._llcrypt registered [ 2520.936849] Key type .llcrypt registered [ 2521.070582] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2523.480830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2523.568735] alg: No test for adler32 (adler32-zlib) [ 2524.932273] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2525.300610] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2527.104770] Key type lgssc registered [ 2528.415258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2536.832622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2536.864560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.754295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.342704] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug +malloc all [ 2549.217589] LustreError: 75938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.253889] LustreError: 75938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2554.345748] LustreError: 75939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.919579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.379373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2561.100444] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug +malloc all [ 2569.347319] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2576.029025] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2582.985029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2583.127550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.180814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2589.262566] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug +malloc all [ 2597.374239] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2610.144244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2610.154141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.172240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2611.681105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2611.684490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2611.689210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.211236] Lustre: server umount lustre-OST0000 complete [ 2621.926430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.927638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.945125] Lustre: Skipped 1 previous similar message [ 2621.953299] Lustre: Skipped 1 previous similar message [ 2626.451497] Lustre: server umount lustre-MDT0000 complete [ 2627.040869] LustreError: 77036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.055825] LustreError: 77036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2630.671666] LustreError: 75919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785370 with bad export cookie 14041456528707795351 [ 2630.682821] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.688324] LustreError: 75919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2631.020558] Lustre: server umount lustre-MDT0001 complete [ 2643.441753] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2646.523345] Key type lgssc unregistered [ 2646.840900] LNet: 79494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.845805] LNetError: 79494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.859910] LNet: Removed LNI 192.168.201.114@tcp [ 2647.879096] Key type .llcrypt unregistered [ 2647.887806] Key type ._llcrypt unregistered [ 2672.776244] Key type ._llcrypt registered [ 2672.778486] Key type .llcrypt registered [ 2672.885184] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2674.485798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2674.516336] alg: No test for adler32 (adler32-zlib) [ 2675.715745] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2675.950114] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2677.599214] Key type lgssc registered [ 2678.862634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2687.680224] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2687.698942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2703.327550] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2708.448434] LustreError: 80884:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.114@tcp: failed processing log, type 4: rc = -110 [ 2735.135418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.572412] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2750.604503] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2759.487781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.760778] Lustre: *** cfs_fail_loc=706, val=0*** [ 2760.312332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.414140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2765.282101] LustreError: 80909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.315379] LustreError: 80909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2765.799972] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2769.401560] LustreError: 80910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.432235] LustreError: 80910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2774.544481] LustreError: 80909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.578326] LustreError: 80909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2775.868212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.233174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2782.043182] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2790.792995] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2797.101982] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2801.140980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2801.152256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.188578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2801.662732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2801.679548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.717341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2806.750195] Lustre: server umount lustre-OST0000 complete [ 2816.998585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2817.015752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.032921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2817.035149] Lustre: Skipped 2 previous similar messages [ 2822.626461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.631270] Lustre: Skipped 3 previous similar messages [ 2827.232981] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2827.520512] Lustre: server umount lustre-MDT0000 complete [ 2827.745302] LustreError: 81974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.760149] LustreError: 81974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2832.547985] LustreError: 80890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785572 with bad export cookie 9863392694482284760 [ 2832.558911] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2832.952458] Lustre: server umount lustre-MDT0001 complete [ 2844.058750] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2847.407123] Key type lgssc unregistered [ 2847.646187] LNet: 84503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2847.660046] LNetError: 84503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2847.677617] LNet: Removed LNI 192.168.201.114@tcp [ 2848.387891] Key type .llcrypt unregistered [ 2848.390860] Key type ._llcrypt unregistered [ 2882.420741] Key type ._llcrypt registered [ 2882.422044] Key type .llcrypt registered [ 2882.520450] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2884.124365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2884.275468] alg: No test for adler32 (adler32-zlib) [ 2885.566760] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 2886.056254] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2887.815223] Key type lgssc registered [ 2890.030386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2899.419731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2899.442954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.150207] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2915.448935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.031366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2921.441535] LustreError: 86608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.475442] LustreError: 86608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2921.619499] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2925.540293] LustreError: 86606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.560530] LustreError: 86606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2930.663653] LustreError: 86611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.676632] LustreError: 86611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2931.609612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.097321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.315120] LustreError: 86611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.335469] LustreError: 86611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2939.642801] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2945.521125] LustreError: 86612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.549365] LustreError: 86612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2950.081209] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2950.123333] Lustre: MGS: Client b5976147-f616-4c7c-affb-0edabe13e846 (at 0@lo) reconnecting [ 2950.152389] LustreError: 86622:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2950.564118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.655189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2957.250675] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2971.112200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.113608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.124690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2971.137722] Lustre: Skipped 1 previous similar message [ 2976.228978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2976.242302] Lustre: Skipped 1 previous similar message [ 2977.421420] Lustre: server umount lustre-OST0000 complete [ 2986.471076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.476292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2986.482385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2986.482394] Lustre: Skipped 1 previous similar message [ 2988.828488] Lustre: server umount lustre-MDT0000 complete [ 2991.586327] LustreError: 88528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2991.609314] LustreError: 88528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2992.925920] Lustre: server umount lustre-MDT0001 complete [ 2997.256704] Lustre: server umount MGS complete [ 3010.755995] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:09:07 (1787785747) [ 3042.345987] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 3052.985172] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3092.171456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.244349] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3108.267979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.673192] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3108.754259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3108.899712] Lustre: lustre-MDT0001: new disk, initializing [ 3108.979521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3109.026198] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3109.042327] Lustre: 93778:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3109.051766] LustreError: 93778:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3113.667309] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3124.181205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.478402] Lustre: lustre-OST0000: new disk, initializing [ 3124.481214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3124.487104] Lustre: 94544:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3124.537109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.076695] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3133.985297] Lustre: 94924:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a8cf6071 x1874629099563008/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787785889 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3144.064977] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3152.650884] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.674912] Lustre: MGS: Client 24d6da9d-0be1-49b2-b15f-29a4dac69053 (at 0@lo) reconnecting [ 3152.746925] Lustre: 93044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3152.768469] LustreError: 93764:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3152.796829] LustreError: 93764:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3152.818430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3152.898989] Lustre: lustre-MDT0000: new disk, initializing [ 3153.059642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3153.102266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3153.155144] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3154.210062] Lustre: 95614:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008d8c63ec x1874629099570176/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787785909 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3157.570425] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3159.348746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3159.371139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3159.497754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3184.613840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.629236] Lustre: Skipped 1 previous similar message [ 3184.633621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3184.636424] Lustre: Skipped 1 previous similar message [ 3189.479391] Lustre: server umount lustre-OST0000 complete [ 3194.847954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3194.849463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.858352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.872499] Lustre: Skipped 2 previous similar messages [ 3194.896861] Lustre: Skipped 1 previous similar message [ 3205.089343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.092562] Lustre: Skipped 3 previous similar messages [ 3209.184193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3209.253162] Lustre: server umount lustre-MDT0000 complete [ 3210.211664] LustreError: 93753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3210.236673] LustreError: 93753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3218.956630] Lustre: server umount lustre-MDT0001 complete [ 3223.040581] Lustre: server umount MGS complete [ 3236.453839] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:12:53 (1787785973) [ 3257.559671] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3261.201451] Key type lgssc unregistered [ 3261.599371] LNet: 98118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3261.616985] LNetError: 98118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3262.699768] LNet: Removed LNI 192.168.201.114@tcp [ 3263.850096] Key type .llcrypt unregistered [ 3263.851725] Key type ._llcrypt unregistered [ 3285.910283] Key type ._llcrypt registered [ 3285.913418] Key type .llcrypt registered [ 3286.155490] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3287.791388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3287.829335] alg: No test for adler32 (adler32-zlib) [ 3288.956746] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3289.264626] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3291.079184] Key type lgssc registered [ 3292.531284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3300.641411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3300.677643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.352733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.665177] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3312.619958] LustreError: 99340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.645996] LustreError: 99340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3317.303692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.729738] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3317.900385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3323.373250] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3332.817431] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3340.279572] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3348.427701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.904899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.461318] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3364.622669] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3372.458692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3387.886930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3387.895211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.929710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3387.943310] Lustre: Skipped 1 previous similar message [ 3391.290448] Lustre: server umount lustre-OST0000 complete [ 3398.117307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.132341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.150686] Lustre: Skipped 2 previous similar messages [ 3402.114494] Lustre: server umount lustre-MDT0000 complete [ 3403.240499] LustreError: 100080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3403.272407] LustreError: 100080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3406.804104] LustreError: 99321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786146 with bad export cookie 3383671070998694197 [ 3406.816700] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3407.183781] Lustre: server umount lustre-MDT0001 complete [ 3421.632399] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3425.009348] Key type lgssc unregistered [ 3425.405458] LNet: 102902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3425.422534] LNetError: 102902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3425.459149] LNet: Removed LNI 192.168.201.114@tcp [ 3426.720434] Key type .llcrypt unregistered [ 3426.724140] Key type ._llcrypt unregistered [ 3448.894843] Key type ._llcrypt registered [ 3448.897760] Key type .llcrypt registered [ 3449.139618] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3450.036216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3450.052325] alg: No test for adler32 (adler32-zlib) [ 3451.702973] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3451.975057] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3453.632994] Key type lgssc registered [ 3455.602817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3463.550032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3463.564584] LustreError: 103984:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3463.615283] LustreError: 103988:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3463.626062] LustreError: 103984:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4951 [ 3463.637983] LustreError: 103984:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3463.647199] LustreError: 103984:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3463.657171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.469872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.986515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.250936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3478.906710] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3485.481398] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing lsmod [ 3489.191737] LustreError: 105194:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3489.197765] LustreError: 105195:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3489.201385] LustreError: 105194:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3489.204605] LustreError: 105194:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3489.210360] LustreError: 105194:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3489.221734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3489.510264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.561192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3494.884310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3494.888075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3494.900141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3495.915756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3495.918364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.922993] Lustre: Skipped 1 previous similar message [ 3495.931245] Lustre: Skipped 1 previous similar message [ 3498.698345] Lustre: server umount lustre-MDT0000 complete [ 3499.064368] LustreError: 104025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3499.079733] LustreError: 104025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3503.112147] LustreError: 103999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786242 with bad export cookie 7561689247759562761 [ 3503.127511] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3503.290541] LustreError: 104456:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003f83db5c x1874629693036800/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3503.406218] Lustre: server umount lustre-MDT0001 complete [ 3518.448230] Lustre: server umount lustre-OST0000 complete [ 3526.909305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3527.337043] LustreError: 106261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.414780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.347633] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3532.776510] LustreError: 106262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.890930] LustreError: 106261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.278618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.604356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.393286] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3551.682454] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3557.025961] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3562.883322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3563.284491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.140223] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3573.628330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3576.485901] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3583.972490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.980558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3583.983211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3583.992060] Lustre: Skipped 1 previous similar message [ 3584.022446] Lustre: Skipped 2 previous similar messages [ 3589.089149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.101929] Lustre: Skipped 1 previous similar message [ 3589.844700] Lustre: server umount lustre-OST0000 complete [ 3594.222806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.231714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3594.236606] Lustre: Skipped 1 previous similar message [ 3594.254460] Lustre: Skipped 1 previous similar message [ 3599.515941] Lustre: server umount lustre-MDT0000 complete [ 3602.657782] LustreError: 106243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786342 with bad export cookie 7561689247759563713 [ 3602.670271] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.677784] LustreError: 106243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3603.302470] Lustre: server umount lustre-MDT0001 complete [ 3617.625735] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3621.298293] Key type lgssc unregistered [ 3621.754563] LNet: 109798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3621.767178] LNetError: 109798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3621.783655] LNet: Removed LNI 192.168.201.114@tcp [ 3623.291816] Key type .llcrypt unregistered [ 3623.298110] Key type ._llcrypt unregistered [ 3647.867260] Key type ._llcrypt registered [ 3647.869620] Key type .llcrypt registered [ 3648.038993] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3649.247579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3649.266282] alg: No test for adler32 (adler32-zlib) [ 3650.397261] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3650.623718] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3652.319807] Key type lgssc registered [ 3653.574694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3661.930142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3661.952802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.530021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.496296] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3674.082336] LustreError: 111215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.102083] LustreError: 111215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3676.724770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.078513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3680.487605] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3687.520694] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3695.078443] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3703.327660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3703.742025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.866827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3710.161276] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3717.684975] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3727.025382] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3730.716087] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3736.906498] Lustre: server umount lustre-OST0000 complete [ 3742.691755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3742.715087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3742.722679] Lustre: Skipped 1 previous similar message [ 3747.811302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3747.824464] Lustre: Skipped 1 previous similar message [ 3750.950369] Lustre: server umount lustre-MDT0000 complete [ 3752.930839] LustreError: 111215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.950346] LustreError: 111215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3753.888311] Lustre: 110525:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786477/real 1787786477] req@0000000066413cd8 x1874629901716352/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787786493 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3753.920424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.928611] Lustre: Skipped 1 previous similar message [ 3753.932946] LustreError: 112480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.943701] LustreError: 112480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3754.583068] LustreError: 111193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786494 with bad export cookie 8580193127491581686 [ 3754.587987] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3754.590428] LustreError: 111193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3755.062484] Lustre: server umount lustre-MDT0001 complete [ 3767.542971] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3770.714617] Key type lgssc unregistered [ 3771.050102] LNet: 114857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3771.066594] LNetError: 114857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3771.087763] LNet: Removed LNI 192.168.201.114@tcp [ 3772.030627] Key type .llcrypt unregistered [ 3772.033352] Key type ._llcrypt unregistered [ 3790.050194] Key type ._llcrypt registered [ 3790.056428] Key type .llcrypt registered [ 3790.223820] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3791.790558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3791.841326] alg: No test for adler32 (adler32-zlib) [ 3793.022612] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3793.375575] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3795.087229] Key type lgssc registered [ 3796.328379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3804.107679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3804.125334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.752035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.103537] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3815.922959] LustreError: 116080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.937994] LustreError: 116080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3820.882861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.025203] LustreError: 116081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3821.384756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3826.237546] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3834.961780] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3841.529197] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3848.063705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3848.554185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.630959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3856.522286] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3866.512792] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3876.320179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3876.326721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3876.336895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.346786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.349432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.374562] Lustre: Skipped 1 previous similar message [ 3881.698948] Lustre: server umount lustre-OST0000 complete [ 3887.586578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.592401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3887.596205] Lustre: Skipped 1 previous similar message [ 3887.614470] Lustre: Skipped 1 previous similar message [ 3891.933581] Lustre: server umount lustre-MDT0000 complete [ 3892.708401] LustreError: 116080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3892.729485] LustreError: 116080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3895.741282] LustreError: 118662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786635 with bad export cookie 13337989470459545522 [ 3895.750710] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3896.108191] Lustre: server umount lustre-MDT0001 complete [ 3909.103387] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3912.024850] Key type lgssc unregistered [ 3912.379277] LNet: 119618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.388609] LNetError: 119618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3912.414792] LNet: Removed LNI 192.168.201.114@tcp [ 3913.424272] Key type .llcrypt unregistered [ 3913.438422] Key type ._llcrypt unregistered [ 3939.880830] Key type ._llcrypt registered [ 3939.883083] Key type .llcrypt registered [ 3939.987765] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3941.374883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3941.416862] alg: No test for adler32 (adler32-zlib) [ 3942.472694] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3942.724813] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3944.415856] Key type lgssc registered [ 3945.556169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3954.050533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3954.074196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.591046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.666810] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3965.930136] LustreError: 121036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.949086] LustreError: 121036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3969.650506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.155477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3975.323730] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3983.871749] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3990.051596] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3997.005967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3997.374242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.518051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 4003.865871] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4011.815477] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4021.952860] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4026.097632] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4026.103201] Lustre: Skipped 2 previous similar messages [ 4030.804889] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4030.813758] Lustre: Skipped 2 previous similar messages [ 4031.085571] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4031.119608] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4036.857499] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.14@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4036.863899] Lustre: Skipped 2 previous similar messages [ 4043.833129] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.14@tcp 192.168.0.[2,10]@tcp [ 4048.574228] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.14/24@tcp in log lustre-MDT0000 [ 4048.576825] Lustre: Skipped 2 previous similar messages [ 4052.738617] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.14/24@tcp [ 4052.753338] Lustre: Skipped 1 previous similar message [ 4064.649665] Lustre: server umount lustre-OST0000 complete [ 4072.430511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4072.430596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.445337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4072.458841] Lustre: Skipped 1 previous similar message [ 4077.537525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.550810] Lustre: Skipped 2 previous similar messages [ 4082.656090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.666598] Lustre: Skipped 1 previous similar message [ 4083.167118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4083.545710] Lustre: server umount lustre-MDT0000 complete [ 4083.618404] Lustre: 120350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786807/real 1787786807] req@00000000e2e97f6d x1874630207923584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787786823 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4083.656101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.678959] LustreError: 121036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.702214] LustreError: 121036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4087.602044] LustreError: 121017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786827 with bad export cookie 8644091674634872969 [ 4087.607990] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4087.615361] LustreError: 121017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4088.118464] Lustre: server umount lustre-MDT0001 complete [ 4101.621157] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4104.533671] Key type lgssc unregistered [ 4104.997518] LNet: 125888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4105.005189] LNetError: 125888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4105.022189] LNet: Removed LNI 192.168.201.114@tcp [ 4106.171853] Key type .llcrypt unregistered [ 4106.174154] Key type ._llcrypt unregistered [ 4137.409795] Key type ._llcrypt registered [ 4137.413836] Key type .llcrypt registered [ 4137.664591] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4139.380535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4139.437060] alg: No test for adler32 (adler32-zlib) [ 4141.137167] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 4141.471046] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4143.247193] Key type lgssc registered [ 4145.588926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4156.404454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4156.462809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4158.053775] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4158.053978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.072788] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.14/24@tcp [ 4164.523326] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4167.667573] LustreError: 127470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.697257] LustreError: 127470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4172.769789] LustreError: 127471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.536873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.907162] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4174.954399] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.14/24@tcp [ 4175.039136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4180.884161] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4189.106522] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4195.491028] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4203.101283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4203.541149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.241761] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4217.802980] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4228.157137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4243.424285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4243.441457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.452225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4243.462640] Lustre: Skipped 1 previous similar message [ 4243.500893] Lustre: Skipped 1 previous similar message [ 4248.546427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4248.562855] Lustre: Skipped 1 previous similar message [ 4253.669235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4253.681130] Lustre: Skipped 1 previous similar message [ 4255.199163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4255.413210] Lustre: server umount lustre-OST0000 complete [ 4261.860573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4261.864718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.878872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4263.904618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.926257] Lustre: server umount lustre-MDT0000 complete [ 4269.025747] LustreError: 129965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.037945] LustreError: 129965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4270.276836] LustreError: 127451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787010 with bad export cookie 3353424673791261937 [ 4270.282617] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4270.292713] LustreError: 127451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4270.636812] Lustre: server umount lustre-MDT0001 complete [ 4284.081825] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4286.764362] Key type lgssc unregistered [ 4287.144738] LNet: 131093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4287.154141] LNetError: 131093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4287.183692] LNet: Removed LNI 192.168.201.114@tcp [ 4288.513451] Key type .llcrypt unregistered [ 4288.517490] Key type ._llcrypt unregistered [ 4317.665829] Key type ._llcrypt registered [ 4317.668662] Key type .llcrypt registered [ 4317.778967] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 4336.938321] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4338.626200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4338.996749] alg: No test for adler32 (adler32-zlib) [ 4339.989617] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 4340.240650] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4342.039180] Key type lgssc registered [ 4343.265225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4380.517260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4380.541505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4381.858450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4381.939370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4382.058631] Lustre: lustre-MDT0000: new disk, initializing [ 4382.165024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.189567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4387.509420] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4399.527380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.718712] Lustre: 134496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4399.851203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4399.856424] Lustre: Skipped 1 previous similar message [ 4399.935546] Lustre: lustre-MDT0001: new disk, initializing [ 4400.052937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4400.143346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4400.175318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4405.079429] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4415.537938] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4422.459047] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4430.622365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4430.918341] Lustre: lustre-OST0000: new disk, initializing [ 4430.920792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4430.926760] Lustre: 136465:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4431.051518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4438.646340] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4439.667054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4439.679227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4439.832096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4449.043503] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4454.126320] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4454.381753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4459.322425] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4459.569097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4465.644027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4465.647332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.656892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4465.665275] Lustre: Skipped 1 previous similar message [ 4465.691871] Lustre: Skipped 1 previous similar message [ 4468.030206] Lustre: server umount lustre-OST0000 complete [ 4475.874133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4475.902147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4475.910029] Lustre: Skipped 1 previous similar message [ 4478.939743] Lustre: server umount lustre-MDT0000 complete [ 4482.744563] LustreError: 134489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787222 with bad export cookie 15111934679069740418 [ 4482.751319] LustreError: 134489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4482.758396] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4483.094476] Lustre: server umount lustre-MDT0001 complete [ 4490.386935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.850341] LustreError: 138681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.886439] LustreError: 138681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4490.986972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.085964] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4496.353573] LustreError: 138682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.460950] LustreError: 138681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.571926] LustreError: 138682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.844746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.667088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4513.927110] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4523.302874] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4530.250228] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4539.068558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4539.637029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.393961] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4554.294913] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4559.979768] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4562.465518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4567.775257] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4568.046815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4577.878038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4578.217450] Lustre: lustre-OST0001: new disk, initializing [ 4578.225151] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4578.238095] Lustre: 142069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4578.404821] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4585.625190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4585.638198] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4585.778067] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4586.451096] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4596.366944] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4596.773341] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4603.353722] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4603.659414] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4609.866045] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88cd4551c800.ost_server_uuid 50 [ 4611.445793] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88cd4551c800.ost_server_uuid in IDLE state after 0 sec [ 4621.800425] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4621.801203] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.802876] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4621.849353] Lustre: Skipped 2 previous similar messages [ 4623.577663] Lustre: server umount lustre-OST0001 complete [ 4626.924634] LustreError: 140507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.945710] LustreError: 140507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4627.937496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4627.955782] LustreError: Skipped 1 previous similar message [ 4627.963272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.993948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4628.012624] Lustre: Skipped 1 previous similar message [ 4632.035726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4632.047771] Lustre: Skipped 2 previous similar messages [ 4637.156850] LustreError: 140508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.176097] LustreError: 140508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4642.271207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4642.276752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4642.304319] Lustre: Skipped 3 previous similar messages [ 4642.442956] Lustre: server umount lustre-OST0000 complete [ 4647.394608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.411451] Lustre: Skipped 2 previous similar messages [ 4653.212903] Lustre: server umount lustre-MDT0000 complete [ 4656.696936] LustreError: 138662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787396 with bad export cookie 15111934679069741300 [ 4656.699567] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4656.715447] LustreError: 138662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4657.304686] Lustre: server umount lustre-MDT0001 complete [ 4680.306804] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4683.969971] Key type lgssc unregistered [ 4684.366784] LNet: 145075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4684.376502] LNetError: 145075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4684.390700] LNet: Removed LNI 192.168.201.114@tcp [ 4685.398655] Key type .llcrypt unregistered [ 4685.400334] Key type ._llcrypt unregistered [ 4734.943175] Key type ._llcrypt registered [ 4734.953745] Key type .llcrypt registered [ 4735.151919] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 4754.507782] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4756.628279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4756.720300] alg: No test for adler32 (adler32-zlib) [ 4757.770258] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 4758.034975] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4759.727174] Key type lgssc registered [ 4761.117665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4802.023672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4802.044307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.425469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4803.468686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4803.617290] Lustre: lustre-MDT0000: new disk, initializing [ 4803.871238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4803.911483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4808.772873] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4821.922460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4822.027068] Lustre: 149238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4822.097452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4822.101136] Lustre: Skipped 1 previous similar message [ 4822.168177] Lustre: lustre-MDT0001: new disk, initializing [ 4822.267581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4822.308324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4822.325838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4828.040879] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4841.775446] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4849.225710] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4858.031308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4858.427695] Lustre: lustre-OST0000: new disk, initializing [ 4858.433766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4858.444043] Lustre: 151207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4858.610562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4864.666345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4864.707626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4864.843824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4866.598663] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4877.268571] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4883.055284] Lustre: Failing over lustre-OST0000 [ 4883.340379] Lustre: server umount lustre-OST0000 complete [ 4900.321747] Lustre: 147456:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787624/real 1787787624] req@000000007654ed6b x1874631062490880/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787787640 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4900.366483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.471492] Lustre: 147459:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787625/real 1787787625] req@000000004ab55dc4 x1874631062491776/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787787641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4901.477186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.498673] Lustre: 147459:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4904.184372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.630195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4904.642278] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4906.655226] Lustre: 147457:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787630/real 1787787630] req@0000000066d6c286 x1874631062492928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787787646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4910.766335] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4911.775207] Lustre: 147457:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787635/real 1787787635] req@0000000066ba2ac7 x1874631062494080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787787651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4911.799675] Lustre: 147457:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4911.918435] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4912.009142] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4912.010713] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4916.823893] Lustre: Failing over lustre-MDT0000 [ 4917.429667] Lustre: server umount lustre-MDT0000 complete [ 4917.862707] Lustre: 147458:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787640/real 1787787640] req@00000000f6892183 x1874631062495232/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787787656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4919.777076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4919.783197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.203497] LustreError: 150264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.231167] LustreError: 150264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4926.432991] LustreError: 149245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.465589] LustreError: 149245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4931.558717] LustreError: 149243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4931.591989] LustreError: 149243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4936.675895] LustreError: 149248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.704579] LustreError: 149248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4937.695656] Lustre: 147457:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787661/real 1787787661] req@00000000a5b42cfb x1874631062508032/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787787677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4937.728360] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4938.333546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4942.817560] LustreError: 149248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.840322] LustreError: 149248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4947.942873] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x23ebab62e9701bd7 [ 4947.953096] Lustre: MGC192.168.201.114@tcp: Connection restored to 0@lo (at 0@lo) [ 4947.962429] Lustre: Skipped 1 previous similar message [ 4948.260318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.330931] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4948.854208] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4953.582266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4953.599524] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4953.659192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4953.910864] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4964.320420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4964.328276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4964.343847] Lustre: Skipped 2 previous similar messages [ 4964.348797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4968.418939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4968.427515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4968.448577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4970.600454] Lustre: server umount lustre-OST0000 complete [ 4979.171119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4979.177784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4979.190042] Lustre: Skipped 1 previous similar message [ 4979.207741] Lustre: Skipped 2 previous similar messages [ 4981.676336] Lustre: server umount lustre-MDT0000 complete [ 4984.289198] LustreError: 149243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.315350] LustreError: 149243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4986.260496] LustreError: 149229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787726 with bad export cookie 2588350852166065111 [ 4986.263459] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4986.278776] LustreError: 149229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4986.848342] Lustre: server umount lustre-MDT0001 complete [ 5000.645559] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5003.796907] Key type lgssc unregistered [ 5004.088628] LNet: 154868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5004.098983] LNetError: 154868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5004.123883] LNet: Removed LNI 192.168.201.114@tcp [ 5005.302152] Key type .llcrypt unregistered [ 5005.306402] Key type ._llcrypt unregistered [ 5047.536744] Key type ._llcrypt registered [ 5047.540434] Key type .llcrypt registered [ 5047.698438] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5049.046254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5049.067801] alg: No test for adler32 (adler32-zlib) [ 5050.417423] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 5050.792828] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5052.551241] Key type lgssc registered [ 5054.098132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5063.742650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5063.754970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.223521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5071.115805] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5074.402651] LustreError: 156389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5074.415094] LustreError: 156389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5079.519989] LustreError: 156390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5079.860924] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5083.960621] LustreError: 156418:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5083.972281] LustreError: 156418:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5083.984281] LustreError: 156418:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 5084.384978] Lustre: server umount lustre-MDT0000 complete [ 5102.451200] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 5115.507506] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5161.675861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.070897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5162.107826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5162.195389] Lustre: lustre-MDT0000: new disk, initializing [ 5162.345996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.380402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5168.156693] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5180.906452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.128501] Lustre: 159727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5181.211142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5181.217580] Lustre: Skipped 1 previous similar message [ 5181.321308] Lustre: lustre-MDT0001: new disk, initializing [ 5181.531076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5181.621183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5181.658426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5186.600164] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5199.747654] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5208.136538] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5217.474388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5218.099744] Lustre: lustre-OST0000: new disk, initializing [ 5218.114900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5218.129418] Lustre: 161695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5218.285743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.563644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5223.590208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5223.723223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5226.439774] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5237.921168] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5243.045308] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5245.304712] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:46:22 (1787787982) [ 5485.024864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.026209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5485.044497] Lustre: Skipped 1 previous similar message [ 5485.105410] Lustre: Skipped 2 previous similar messages [ 5490.157447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5490.172973] Lustre: Skipped 2 previous similar messages [ 5495.264924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5495.275759] Lustre: Skipped 2 previous similar messages [ 5498.335217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5498.649579] Lustre: server umount lustre-MDT0000 complete [ 5500.385118] LustreError: 159732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5500.417097] LustreError: 159732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5502.904377] LustreError: 159717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788242 with bad export cookie 6349201409961185159 [ 5502.912928] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5502.921423] LustreError: 159717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5505.503835] LustreError: 159734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.504337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5505.553536] Lustre: Skipped 1 previous similar message [ 5505.579115] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5512.672724] LustreError: 159732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.674199] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5512.692700] LustreError: 159732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5512.702711] Lustre: Skipped 1 previous similar message [ 5517.279150] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5517.742135] Lustre: server umount lustre-MDT0001 complete [ 5536.223136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5536.523515] Lustre: server umount lustre-OST0000 complete [ 5548.030330] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 5557.942746] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5599.393446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.666338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5599.727370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5599.851437] Lustre: lustre-MDT0000: new disk, initializing [ 5599.931166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.959123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5605.449904] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5616.889454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5617.045397] Lustre: 166364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5617.103416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5617.107033] Lustre: Skipped 1 previous similar message [ 5617.162711] Lustre: lustre-MDT0001: new disk, initializing [ 5617.246366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5617.274653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5617.293821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5622.320375] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5632.049794] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5638.534636] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5646.022319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5646.371492] Lustre: lustre-OST0000: new disk, initializing [ 5646.377265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5646.387307] Lustre: 168329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5646.545208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.242347] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5656.632887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5656.643444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5656.773913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5662.920734] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5668.262681] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5668.557238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5673.471932] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5673.762693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5677.540187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5677.548343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.559784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5677.599320] Lustre: Skipped 1 previous similar message [ 5682.070478] Lustre: server umount lustre-OST0000 complete [ 5687.778165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5687.797695] Lustre: Skipped 1 previous similar message [ 5692.178371] Lustre: server umount lustre-MDT0000 complete [ 5692.899394] LustreError: 166372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5692.941695] LustreError: 166372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5696.391964] LustreError: 166356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788436 with bad export cookie 6349201409961343989 [ 5696.411190] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5696.735776] Lustre: server umount lustre-MDT0001 complete [ 5706.346683] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:54:03 (1787788443) [ 5724.274765] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 5736.315868] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5773.115971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.314645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5773.352149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5773.424733] Lustre: lustre-MDT0000: new disk, initializing [ 5773.503465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5773.520110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5778.447316] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5790.084475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.356748] Lustre: 173084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5790.652059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5794.345945] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5804.195933] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5810.356821] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5818.018476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5818.299431] Lustre: lustre-OST0000: new disk, initializing [ 5818.303638] Lustre: Skipped 1 previous similar message [ 5818.308472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5818.314221] Lustre: Skipped 2 previous similar messages [ 5818.319959] Lustre: 175063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5825.478902] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5825.619796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5825.637872] Lustre: Skipped 1 previous similar message [ 5825.655753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5825.847139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5835.891779] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5846.501971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.502836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5846.511233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5846.511242] Lustre: Skipped 3 previous similar messages [ 5846.530295] Lustre: Skipped 1 previous similar message [ 5846.566040] LustreError: Skipped 2 previous similar messages [ 5850.240947] Lustre: server umount lustre-OST0000 complete [ 5856.738480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.773631] Lustre: Skipped 1 previous similar message [ 5860.886584] Lustre: server umount lustre-MDT0000 complete [ 5861.859870] LustreError: 173092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.888891] LustreError: 173092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5865.259986] LustreError: 175880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788605 with bad export cookie 6349201409961344871 [ 5865.263473] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5865.271311] LustreError: 175880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5865.781237] Lustre: server umount lustre-MDT0001 complete [ 5874.812539] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:56:52 (1787788612) [ 5891.608676] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 5903.734330] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5940.699306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.999418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5941.038443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5941.174408] Lustre: lustre-MDT0000: new disk, initializing [ 5941.269793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5941.278896] Lustre: Skipped 2 previous similar messages [ 5941.305973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5945.951627] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5956.979901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5957.137769] Lustre: 179389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5957.546505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5962.629979] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5972.051313] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5978.118481] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5985.947490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5986.145081] Lustre: 181364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5993.078434] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5993.527122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5993.638987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6001.454190] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6008.801464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6008.818228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6008.848765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6008.857771] Lustre: Skipped 3 previous similar messages [ 6014.442513] LustreError: 181343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.450461] LustreError: 181343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6014.794871] Lustre: server umount lustre-OST0000 complete [ 6019.058342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6019.075920] LustreError: Skipped 1 previous similar message [ 6033.375136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6033.578432] Lustre: server umount lustre-MDT0000 complete [ 6036.754562] LustreError: 179379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788776 with bad export cookie 6349201409961345956 [ 6036.758496] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6036.762733] LustreError: 179379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6037.021706] Lustre: server umount lustre-MDT0001 complete [ 6049.124288] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6052.719203] Key type lgssc unregistered [ 6053.088654] LNet: 183224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6053.095436] LNetError: 183224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6053.109909] LNet: Removed LNI 192.168.201.114@tcp [ 6054.375497] Key type .llcrypt unregistered [ 6054.377434] Key type ._llcrypt unregistered [ 6078.572339] Key type ._llcrypt registered [ 6078.575019] Key type .llcrypt registered [ 6078.689986] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6079.783226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6079.863461] alg: No test for adler32 (adler32-zlib) [ 6080.940454] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 6081.158209] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6082.855245] Key type lgssc registered [ 6084.190353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6092.430283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6092.447568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.006175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.908253] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6103.522670] LustreError: 184642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.547814] LustreError: 184642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6108.644750] LustreError: 184643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.519243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6109.857391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6115.963277] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6125.606831] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6131.464441] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6139.213092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6139.663217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6146.791092] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6155.773526] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6163.333495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6185.952441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.967273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.995377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6189.024342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6189.028386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6189.039798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6189.061430] Lustre: Skipped 1 previous similar message [ 6191.319288] Lustre: server umount lustre-OST0000 complete [ 6197.227638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6197.241396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6197.261602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.264341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.266367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.928606] Lustre: server umount lustre-MDT0000 complete [ 6204.388383] LustreError: 184639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.431244] LustreError: 184639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6206.156828] LustreError: 184623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788945 with bad export cookie 3688616162989520436 [ 6206.165029] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6206.541383] Lustre: server umount lustre-MDT0001 complete [ 6219.150469] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6222.403893] Key type lgssc unregistered [ 6222.861745] LNet: 188217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6222.868797] LNetError: 188217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6222.897380] LNet: Removed LNI 192.168.201.114@tcp [ 6223.920036] Key type .llcrypt unregistered [ 6223.922897] Key type ._llcrypt unregistered [ 6253.116311] Key type ._llcrypt registered [ 6253.118925] Key type .llcrypt registered [ 6253.263144] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6254.578567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6254.615541] alg: No test for adler32 (adler32-zlib) [ 6255.844116] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 6256.151677] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6257.863172] Key type lgssc registered [ 6259.514235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6267.726781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6267.758469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.294633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.453543] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6279.649062] LustreError: 189635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.686736] LustreError: 189635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6283.598124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6284.156703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6289.703785] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6298.290065] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6304.630474] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6312.420487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6312.928466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6315.005215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6320.063370] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6328.980923] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6336.914967] Lustre: lustre-OST0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 6338.528285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6338.542825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6338.547308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6338.567682] Lustre: Skipped 1 previous similar message [ 6338.583358] Lustre: Skipped 1 previous similar message [ 6341.989168] LustreError: 191458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.208584] Lustre: server umount lustre-OST0000 complete [ 6350.683279] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88cd44a63800.ost_server_uuid 50 [ 6352.526719] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6369.248908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.259274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6369.273438] Lustre: Skipped 1 previous similar message [ 6372.841205] Lustre: server umount lustre-MDT0000 complete [ 6374.368365] LustreError: 189635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6374.381601] LustreError: 189635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6376.632444] LustreError: 189616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789116 with bad export cookie 18180299413633569438 [ 6376.636961] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6376.643371] LustreError: 189616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6377.194459] Lustre: server umount lustre-MDT0001 complete [ 6385.919528] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:05:23 (1787789123) [ 6392.923736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.548247] LustreError: 193551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.607683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6398.945452] LustreError: 193552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.234314] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6408.461834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6409.100532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6414.480718] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6422.420987] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6428.199464] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6434.674456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6434.975695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6439.104668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6441.655895] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6448.689869] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6456.140774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.326470] Lustre: lustre-OST0001: new disk, initializing [ 6456.331650] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6456.340701] Lustre: 196392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6456.459899] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6462.931895] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6465.128739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6465.145492] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6465.244262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6473.086563] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6479.233537] Lustre: lustre-OST0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 6479.247426] Lustre: Skipped 1 previous similar message [ 6479.329093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6479.333946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.353252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6482.356497] Lustre: server umount lustre-OST0000 complete [ 6484.321794] LustreError: 195372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6484.341608] LustreError: 195372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6488.485946] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6488.648311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6493.646383] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6493.963588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6501.352786] LustreError: 195371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6501.393362] LustreError: 195371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6511.597229] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.606757] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6511.610888] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6511.610896] Lustre: Skipped 3 previous similar messages [ 6511.634996] Lustre: Skipped 2 previous similar messages [ 6511.655416] LustreError: Skipped 1 previous similar message [ 6517.229415] Lustre: server umount lustre-OST0001 complete [ 6526.947065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.947808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6526.957224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.957239] Lustre: Skipped 3 previous similar messages [ 6528.714752] Lustre: server umount lustre-MDT0000 complete [ 6533.066604] LustreError: 196391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789272 with bad export cookie 18180299413633570509 [ 6533.068813] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6533.079637] LustreError: 196391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6533.305551] Lustre: server umount lustre-MDT0001 complete [ 6558.058352] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:08:15 (1787789295) [ 6565.075830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.299793] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6565.331534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6565.512277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6570.617201] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6580.656019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6580.806506] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6580.834804] Lustre: 199834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6585.912358] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6593.840891] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6599.878362] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6607.841996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6608.087631] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6614.457519] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6616.726466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6622.317330] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6629.288351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6629.385852] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6629.764260] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6629.780769] Lustre: Skipped 2 previous similar messages [ 6636.084626] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6638.752595] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6643.289598] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6647.266900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6647.276808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.287353] Lustre: Skipped 1 previous similar message [ 6647.293733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6647.298779] Lustre: Skipped 1 previous similar message [ 6652.442342] Lustre: server umount lustre-OST0000 complete [ 6654.303994] LustreError: 201683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6654.335236] LustreError: 201683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6669.793954] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6669.802490] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.808189] LustreError: Skipped 2 previous similar messages [ 6669.837783] Lustre: Skipped 2 previous similar messages [ 6674.586968] Lustre: server umount lustre-OST0001 complete [ 6680.033983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6680.039488] Lustre: Skipped 8 previous similar messages [ 6692.319276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6692.746940] Lustre: server umount lustre-MDT0000 complete [ 6697.346430] LustreError: 199828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789437 with bad export cookie 18180299413633571916 [ 6697.352116] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6697.355139] LustreError: 199828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6697.761634] Lustre: server umount lustre-MDT0001 complete [ 6725.079277] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:11:02 (1787789462) [ 6746.626961] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 6758.850035] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6795.868975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.091520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6796.109080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6796.188475] Lustre: lustre-MDT0000: new disk, initializing [ 6796.292218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6796.316748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6799.914494] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6808.812388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6808.987100] Lustre: 207932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6809.045857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6809.049773] Lustre: Skipped 1 previous similar message [ 6809.121779] Lustre: lustre-MDT0001: new disk, initializing [ 6809.266179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6809.283245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6814.035272] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6822.291339] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6827.507781] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6833.914860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6834.136546] Lustre: lustre-OST0000: new disk, initializing [ 6834.139213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6834.143639] Lustre: 209901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6840.024665] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6843.975760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6843.986151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6844.105923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6848.339314] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6852.696422] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6852.876319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6857.029723] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6857.245083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6859.745362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.749115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6859.753864] Lustre: Skipped 3 previous similar messages [ 6859.764330] Lustre: Skipped 4 previous similar messages [ 6865.356537] Lustre: server umount lustre-OST0000 complete [ 6875.108311] LustreError: 207938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6875.143108] LustreError: 207938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6875.286726] Lustre: server umount lustre-MDT0000 complete [ 6879.478122] LustreError: 207922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789619 with bad export cookie 18180299413633573694 [ 6879.491372] LustreError: 207922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6879.501379] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6879.832327] Lustre: server umount lustre-MDT0001 complete [ 6889.907529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.920564] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6902.388704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.974906] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6914.086879] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6921.268760] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6929.481359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.889709] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6944.202360] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6949.163504] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6949.426463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6954.569803] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6954.730673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6958.566694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.574877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6958.581139] Lustre: Skipped 2 previous similar messages [ 6958.598581] LustreError: Skipped 1 previous similar message [ 6963.108281] Lustre: server umount lustre-OST0000 complete [ 6969.536041] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6974.142813] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7002.267929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7009.228992] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7015.795174] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7022.600454] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 7024.050588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7024.077184] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7024.079080] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7029.216282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7029.222132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.243740] Lustre: Skipped 8 previous similar messages [ 7033.387304] Lustre: server umount lustre-OST0000 complete [ 7043.065868] Lustre: server umount lustre-MDT0000 complete [ 7044.581758] LustreError: 212115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7044.592488] LustreError: 212115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7047.209681] LustreError: 212101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789786 with bad export cookie 18180299413633574583 [ 7047.221357] LustreError: 212101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7047.230259] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7047.608750] Lustre: server umount lustre-MDT0001 complete [ 7055.631753] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:16:33 (1787789793) [ 7062.220826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7062.840712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.847150] Lustre: Skipped 6 previous similar messages [ 7067.624729] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7076.659199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.239589] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7088.520193] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7093.689947] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7099.702081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7105.633613] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7112.520196] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7117.127805] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7117.427959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7122.321422] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7122.573550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7128.792599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7128.996913] Lustre: lustre-OST0001: new disk, initializing [ 7129.001534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7129.007364] Lustre: 221452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7134.864312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7134.878296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7134.979500] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7135.036554] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7143.460597] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7147.570430] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7147.725984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7151.105105] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7151.367254] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7155.682296] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.724710] Lustre: Skipped 6 previous similar messages [ 7159.112496] Lustre: server umount lustre-OST0001 complete [ 7165.912211] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7166.180884] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7169.567927] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7169.820740] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7198.284712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7203.823467] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7203.845760] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7203.849938] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7203.863201] Lustre: Skipped 2 previous similar messages [ 7204.638608] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7211.705729] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7213.322172] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7219.168153] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7219.179553] LustreError: Skipped 3 previous similar messages [ 7221.259392] Lustre: server umount lustre-OST0001 complete [ 7243.900483] Lustre: server umount lustre-OST0000 complete [ 7253.639662] Lustre: server umount lustre-MDT0000 complete [ 7257.541380] LustreError: 218063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789997 with bad export cookie 18180299413633575542 [ 7257.545311] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7263.845422] Lustre: server umount lustre-MDT0001 complete [ 7285.306314] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:20:22 (1787790022) [ 7292.427997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7293.059975] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7293.109581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7298.424480] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7310.119945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.361609] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7310.419353] Lustre: 226695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7316.684288] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7324.263250] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7328.373386] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7333.378930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.539955] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7338.962382] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7344.594559] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7352.831465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7352.881456] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7358.706978] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7364.742920] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7366.034848] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7370.161691] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7370.391622] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7373.832376] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7374.006724] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7378.571705] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88cd48a50000.ost_server_uuid 50 [ 7380.026724] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88cd48a50000.ost_server_uuid in IDLE state after 0 sec [ 7380.915151] Lustre: Permanently deactivating lustre-OST0001 [ 7380.927832] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7384.247824] Lustre: Permanently reactivating lustre-OST0001 [ 7390.189903] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7390.202044] Lustre: Skipped 11 previous similar messages [ 7393.957850] Lustre: server umount lustre-OST0001 complete [ 7395.301843] LustreError: 228540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.335593] LustreError: 228540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 7412.191144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7412.327674] Lustre: server umount lustre-OST0000 complete [ 7415.776167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7415.784108] Lustre: Skipped 11 previous similar messages [ 7421.894756] Lustre: server umount lustre-MDT0000 complete [ 7425.174914] LustreError: 226686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790164 with bad export cookie 18180299413633576830 [ 7425.190644] LustreError: 226686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7425.204254] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7425.423261] Lustre: server umount lustre-MDT0001 complete [ 7447.558869] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:23:04 (1787790184) [ 7454.592993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.877397] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7454.896994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7454.904588] Lustre: Skipped 5 previous similar messages [ 7458.763631] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7467.484532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7467.572107] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7467.600304] Lustre: 233387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7471.876654] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7478.487353] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7483.542813] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7490.701451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7490.912078] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7490.956229] Lustre: Permanently deactivating lustre-OST0000 [ 7490.975490] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7495.898301] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7497.796402] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7504.455360] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7510.922649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7510.987026] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7516.035367] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7518.325989] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7522.622768] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7524.844212] Lustre: Permanently reactivating lustre-OST0000 [ 7524.880979] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7524.888707] Lustre: Skipped 2 previous similar messages [ 7526.186483] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7526.191888] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7526.205234] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7526.248243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7526.283135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7538.381565] Lustre: server umount lustre-OST0001 complete [ 7538.656514] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7538.667511] LustreError: Skipped 4 previous similar messages [ 7547.565221] Lustre: server umount lustre-OST0000 complete [ 7557.713581] Lustre: server umount lustre-MDT0000 complete [ 7561.262179] LustreError: 233378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790301 with bad export cookie 18180299413633578489 [ 7561.266167] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7561.273061] LustreError: 233378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7561.672698] Lustre: server umount lustre-MDT0001 complete [ 7570.555796] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7572.343832] Key type lgssc unregistered [ 7572.523571] LNet: 238249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7572.527700] LNetError: 238249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7572.542595] LNet: Removed LNI 192.168.201.114@tcp [ 7573.021190] Key type .llcrypt unregistered [ 7573.023099] Key type ._llcrypt unregistered [ 7596.114755] Key type ._llcrypt registered [ 7596.119462] Key type .llcrypt registered [ 7596.203937] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7596.903491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7596.962713] alg: No test for adler32 (adler32-zlib) [ 7598.081471] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 7598.392703] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7600.023157] Key type lgssc registered [ 7600.896504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7609.430140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7609.462790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7610.990088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.992085] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7620.582678] LustreError: 239717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7620.599913] LustreError: 239717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7621.642252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7621.781488] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7621.805386] Lustre: 239705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7621.813615] Lustre: Permanently deactivating lustre-MDT0001 [ 7621.825970] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7622.157055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7626.363446] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7626.767688] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7626.771703] LustreError: 239748:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7626.785604] LustreError: 239748:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7626.796964] LustreError: 239748:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7633.143861] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7638.619308] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7644.640555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7644.965426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.018276] LustreError: 241558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.034189] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7650.815592] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7652.337030] LustreError: 241696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.399826] LustreError: 241696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.423880] LustreError: 241696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7655.427344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7657.290906] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7660.519369] LustreError: 241558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.537635] LustreError: 241558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7663.949550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7664.230835] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7669.230431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7669.236741] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7670.051519] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7676.075226] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7679.603984] Lustre: Permanently reactivating lustre-MDT0001 [ 7679.618652] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7679.622593] Lustre: Skipped 2 previous similar messages [ 7685.000493] Lustre: lustre-MDT0001: Client 32694015-cf5c-4864-8fe2-f2fd00bcf996 (at 192.168.201.14@tcp) reconnecting [ 7689.581303] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7690.731232] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7695.108722] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7695.303288] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7698.958569] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7699.131967] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7701.198471] Lustre: Permanently deactivating lustre-MDT0001 [ 7701.214659] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7701.222042] Lustre: Skipped 2 previous similar messages [ 7707.291657] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7718.408632] Lustre: server umount lustre-MDT0000 complete [ 7719.395054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7719.406386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7719.431605] LustreError: 240462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7720.420247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.442801] Lustre: Skipped 2 previous similar messages [ 7721.366382] LustreError: 243267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790461 with bad export cookie 16122982225865529985 [ 7721.374253] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7721.378575] LustreError: 243267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7721.568206] Lustre: server umount lustre-MDT0001 complete [ 7724.090221] Lustre: server umount lustre-OST0000 complete [ 7734.971341] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:27:52 (1787790472) [ 7744.779096] Lustre: server umount lustre-OST0001 complete [ 7751.338576] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 7760.247151] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7789.687868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.871301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7789.874912] Lustre: Skipped 2 previous similar messages [ 7789.886304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7789.980959] Lustre: lustre-MDT0000: new disk, initializing [ 7790.098565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.130787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7794.092594] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7803.431335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.574775] Lustre: 247991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7803.666699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7803.671272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7803.675447] Lustre: Skipped 1 previous similar message [ 7803.772887] Lustre: lustre-MDT0001: new disk, initializing [ 7803.883131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7803.918421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7803.932185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7808.021828] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7816.947988] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7822.672440] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7829.023089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.387986] Lustre: lustre-OST0000: new disk, initializing [ 7829.390796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7829.396325] Lustre: 249963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7829.497406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7835.633783] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7838.279301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7838.296482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7838.390874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7844.492403] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7846.920984] LustreError: 247998:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7855.375496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7855.554620] Lustre: lustre-OST0001: new disk, initializing [ 7855.558741] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7855.563224] Lustre: 251075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7856.952614] LustreError: 247998:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7856.999549] LustreError: 247997:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7861.356114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7861.375877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7861.519435] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7861.755586] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7867.041385] LustreError: 247997:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7871.893454] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7873.454622] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7877.109457] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7877.120111] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.137327] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7877.143736] Lustre: Skipped 1 previous similar message [ 7881.375682] Lustre: server umount lustre-OST0001 complete [ 7882.219643] LustreError: 249947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.237163] LustreError: 249947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7897.572894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7897.581602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7897.590066] Lustre: Skipped 2 previous similar messages [ 7897.599223] Lustre: Skipped 1 previous similar message [ 7902.691097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7902.696809] Lustre: Skipped 1 previous similar message [ 7903.727355] Lustre: server umount lustre-OST0000 complete [ 7907.808635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7907.810566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7907.829791] Lustre: Skipped 1 previous similar message [ 7907.874763] Lustre: Skipped 1 previous similar message [ 7913.589717] Lustre: server umount lustre-MDT0000 complete [ 7917.023244] LustreError: 251074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790656 with bad export cookie 16122982225865532799 [ 7917.026818] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7917.032607] LustreError: 251074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7931.359523] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7931.755918] Lustre: server umount lustre-MDT0001 complete [ 7942.203459] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7945.199432] Key type lgssc unregistered [ 7945.497719] LNet: 253115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7945.503719] LNetError: 253115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7945.519420] LNet: Removed LNI 192.168.201.114@tcp [ 7946.441787] Key type .llcrypt unregistered [ 7946.443341] Key type ._llcrypt unregistered [ 7980.203563] Key type ._llcrypt registered [ 7980.205984] Key type .llcrypt registered [ 7980.329227] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7981.390736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7981.414821] alg: No test for adler32 (adler32-zlib) [ 7982.493685] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 7982.787897] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7984.431653] Key type lgssc registered [ 7985.517807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7991.991541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7992.005930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.199079] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7993.225139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7993.424046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7997.179224] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8004.133123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.217878] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8004.238585] Lustre: 255305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8004.415861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8008.251588] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8015.311430] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8019.611861] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8025.428633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8025.593688] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8025.883920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.076856] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8034.974305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8038.620436] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8091.104301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8091.112571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8091.143390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8091.618761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8091.623194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8091.668706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8093.527782] Lustre: server umount lustre-OST0000 complete [ 8096.734660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8105.887578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8106.126206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8111.277782] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8111.293888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8111.309624] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8111.324684] Lustre: Skipped 1 previous similar message [ 8112.817491] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8118.562829] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8157.665865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.671744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8157.675196] Lustre: Skipped 1 previous similar message [ 8157.685467] Lustre: Skipped 1 previous similar message [ 8162.718811] Lustre: server umount lustre-OST0000 complete [ 8167.911415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.918178] Lustre: Skipped 1 previous similar message [ 8167.925783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8172.669088] Lustre: server umount lustre-MDT0000 complete [ 8175.948856] LustreError: 255296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790915 with bad export cookie 14117855020200360220 [ 8175.956878] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8176.188558] Lustre: server umount lustre-MDT0001 complete [ 8186.065991] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8188.323481] Key type lgssc unregistered [ 8188.575847] LNet: 260729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8188.590407] LNetError: 260729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8189.669267] LNet: Removed LNI 192.168.201.114@tcp [ 8190.332216] Key type .llcrypt unregistered [ 8190.334116] Key type ._llcrypt unregistered [ 8209.469267] Key type ._llcrypt registered [ 8209.470894] Key type .llcrypt registered [ 8209.572975] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8210.682947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8210.754086] alg: No test for adler32 (adler32-zlib) [ 8211.729343] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 8211.965326] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8213.607172] Key type lgssc registered [ 8214.794539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8222.338554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8222.359921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8224.005574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.062238] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8233.442771] LustreError: 262126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8233.471772] LustreError: 262126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8236.729518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8237.168777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8240.710782] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8246.858862] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8251.277730] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8258.323542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8258.640873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8264.906740] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8268.915204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8271.720461] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8292.832434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8292.838917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8292.852731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8294.880114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8294.885271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.894403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8294.917511] Lustre: Skipped 1 previous similar message [ 8297.957539] Lustre: server umount lustre-OST0000 complete [ 8303.588135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8303.600525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8303.627229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8310.243038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8310.250259] Lustre: Skipped 3 previous similar messages [ 8315.360601] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8315.370207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8315.385941] Lustre: Skipped 1 previous similar message [ 8315.525338] Lustre: server umount lustre-MDT0000 complete [ 8319.468112] LustreError: 263221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8319.497690] LustreError: 263221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8319.705462] LustreError: 262860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791059 with bad export cookie 1801599682556572830 [ 8319.720170] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8320.112155] Lustre: server umount lustre-MDT0001 complete [ 8331.052340] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8333.971704] Key type lgssc unregistered [ 8334.297867] LNet: 266433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8334.312782] LNetError: 266433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8334.331330] LNet: Removed LNI 192.168.201.114@tcp [ 8335.186628] Key type .llcrypt unregistered [ 8335.191558] Key type ._llcrypt unregistered [ 8352.921035] Key type ._llcrypt registered [ 8352.922551] Key type .llcrypt registered [ 8352.984821] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8354.025948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8354.084608] alg: No test for adler32 (adler32-zlib) [ 8355.184801] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 8355.512082] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8357.239250] Key type lgssc registered [ 8358.459276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8366.154120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8366.168609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8367.618091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8372.566278] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8377.827642] LustreError: 267663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.850110] LustreError: 267663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8380.343503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.644636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8384.528815] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8391.178455] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8396.430597] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8402.448548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8402.744033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.537672] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8413.002370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8415.210569] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8426.464961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8426.475308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8426.508172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8428.514634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.517380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8428.536232] Lustre: Skipped 1 previous similar message [ 8431.306303] Lustre: server umount lustre-OST0000 complete [ 8437.216733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8437.222490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.242098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8441.187623] Lustre: server umount lustre-MDT0000 complete [ 8443.872676] LustreError: 267664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.884592] LustreError: 267664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8444.264624] LustreError: 267643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791184 with bad export cookie 11227573304866837158 [ 8444.275025] LustreError: 267643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8444.275582] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8444.635412] Lustre: server umount lustre-MDT0001 complete [ 8456.643680] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8459.169710] Key type lgssc unregistered [ 8459.461749] LNet: 271342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8459.468969] LNetError: 271342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8459.483340] LNet: Removed LNI 192.168.201.114@tcp [ 8460.328985] Key type .llcrypt unregistered [ 8460.331694] Key type ._llcrypt unregistered [ 8484.189323] Key type ._llcrypt registered [ 8484.193205] Key type .llcrypt registered [ 8484.283792] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8485.354519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8485.538854] alg: No test for adler32 (adler32-zlib) [ 8486.663714] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 8486.936363] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8488.615809] Key type lgssc registered [ 8489.894339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8497.943632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8497.972762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8499.408325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8503.043680] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8509.926507] LustreError: 272760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8509.941356] LustreError: 272760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8510.598747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.990195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8515.179782] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8521.675789] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8527.305214] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8533.475311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8533.826580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.035280] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8545.148174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8546.383105] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8566.753731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8566.765320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8566.773928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8570.863228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.880664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8570.884260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8570.911829] Lustre: Skipped 1 previous similar message [ 8572.649157] Lustre: server umount lustre-OST0000 complete [ 8575.969045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8575.977676] Lustre: Skipped 1 previous similar message [ 8575.979706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8575.984187] Lustre: Skipped 1 previous similar message [ 8581.090925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8581.102776] Lustre: Skipped 1 previous similar message [ 8581.776628] Lustre: server umount lustre-MDT0000 complete [ 8584.721069] LustreError: 272740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791324 with bad export cookie 3872740494126340820 [ 8584.731107] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8584.731542] LustreError: 272740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8584.994945] Lustre: server umount lustre-MDT0001 complete [ 8594.161675] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8596.479340] Key type lgssc unregistered [ 8596.718336] LNet: 277066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8596.726414] LNetError: 277066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8596.748883] LNet: Removed LNI 192.168.201.114@tcp [ 8597.391156] Key type .llcrypt unregistered [ 8597.393396] Key type ._llcrypt unregistered [ 8613.089664] Key type ._llcrypt registered [ 8613.093859] Key type .llcrypt registered [ 8613.205409] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8614.177493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8614.222030] alg: No test for adler32 (adler32-zlib) [ 8615.296349] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 8615.547288] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8617.207260] Key type lgssc registered [ 8618.213802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8623.978764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8623.992236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8625.317482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8627.622385] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8635.756743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.874516] LustreError: 278301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8635.899380] LustreError: 278301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8636.192490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8641.739452] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8649.961809] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8655.462522] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8661.429940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8661.806324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8666.992898] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8671.021956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8673.003347] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8691.174087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8691.181402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.190138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8691.693047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8691.710942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.730737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8696.802667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.570545] Lustre: server umount lustre-OST0000 complete [ 8701.926421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8701.953695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8701.958567] Lustre: Skipped 2 previous similar messages [ 8707.048380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8707.055873] Lustre: Skipped 1 previous similar message [ 8715.231126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8715.487984] Lustre: server umount lustre-MDT0000 complete [ 8717.282489] LustreError: 278301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8717.303171] LustreError: 278301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8718.013709] LustreError: 278266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791457 with bad export cookie 4489994986239129702 [ 8718.023124] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8718.030976] LustreError: 278266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8718.432746] Lustre: server umount lustre-MDT0001 complete [ 8727.478948] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8729.458386] Key type lgssc unregistered [ 8729.816382] LNet: 282026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8729.820887] LNetError: 282026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8729.835056] LNet: Removed LNI 192.168.201.114@tcp [ 8730.412417] Key type .llcrypt unregistered [ 8730.414034] Key type ._llcrypt unregistered [ 8743.053336] Key type ._llcrypt registered [ 8743.056967] Key type .llcrypt registered [ 8743.152389] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8760.771548] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 8774.632719] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8775.918353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8775.983066] alg: No test for adler32 (adler32-zlib) [ 8776.972974] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 8777.155902] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8778.775237] Key type lgssc registered [ 8779.826959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8810.509917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8810.537416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8811.881506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8811.922704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8811.984726] Lustre: lustre-MDT0000: new disk, initializing [ 8812.039136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8812.051567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8816.325575] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8825.643054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8825.781433] Lustre: 285580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8825.851131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8825.853556] Lustre: Skipped 1 previous similar message [ 8825.916876] Lustre: lustre-MDT0001: new disk, initializing [ 8825.998431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8826.030284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8826.046083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8829.943705] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8838.377430] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8843.192629] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8848.665668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8848.917358] Lustre: lustre-OST0000: new disk, initializing [ 8848.920641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8848.928415] Lustre: 287545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8849.038420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8854.678506] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8856.609073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8856.615555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8856.676169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8862.851577] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8866.870958] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8867.060900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8871.059929] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8871.225518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8877.032610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8877.048723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.072613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8877.085131] Lustre: Skipped 1 previous similar message [ 8879.247458] Lustre: server umount lustre-OST0000 complete [ 8887.265697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.270070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8887.285520] Lustre: Skipped 2 previous similar messages [ 8887.308632] Lustre: Skipped 1 previous similar message [ 8889.267942] Lustre: server umount lustre-MDT0000 complete [ 8892.395744] LustreError: 287554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.420106] LustreError: 287554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8892.824973] LustreError: 285571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791632 with bad export cookie 1872638558314326446 [ 8892.826869] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8892.850035] LustreError: 285571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8893.246934] Lustre: server umount lustre-MDT0001 complete [ 8902.075691] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:47:19 (1787791639) [ 8908.081389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8908.498143] LustreError: 289957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8908.587745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8912.878929] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8913.897386] LustreError: 289958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8919.010531] LustreError: 289957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8920.273603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8920.631504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8924.429922] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8931.478230] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8936.623967] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8942.486618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8942.753032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8947.792608] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8954.834806] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8964.583404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8964.598881] LustreError: Skipped 1 previous similar message [ 8964.605215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8964.625239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8964.629258] Lustre: Skipped 1 previous similar message [ 8969.716850] Lustre: server umount lustre-OST0000 complete [ 8974.818602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8974.823218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8974.832113] Lustre: Skipped 2 previous similar messages [ 8974.854385] Lustre: Skipped 1 previous similar message [ 8978.959401] Lustre: server umount lustre-MDT0000 complete [ 8979.940344] LustreError: 289957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8979.964936] LustreError: 289957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8982.125665] LustreError: 289939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791721 with bad export cookie 1872638558314327321 [ 8982.136775] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.371211] Lustre: server umount lustre-MDT0001 complete [ 8991.429527] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8993.800708] Key type lgssc unregistered [ 8994.038762] LNet: 293495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8994.046825] LNetError: 293495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8994.062300] LNet: Removed LNI 192.168.201.114@tcp [ 8994.829058] Key type .llcrypt unregistered [ 8994.830684] Key type ._llcrypt unregistered [ 9018.336766] Key type ._llcrypt registered [ 9018.339282] Key type .llcrypt registered [ 9018.418514] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 9032.640516] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9033.671839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9033.727729] alg: No test for adler32 (adler32-zlib) [ 9034.718229] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 9034.923884] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 9036.583176] Key type lgssc registered [ 9037.493037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9077.739373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9077.763272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9079.043996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9079.063539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9079.145205] Lustre: lustre-MDT0000: new disk, initializing [ 9079.195257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9079.207906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9083.365121] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9093.311993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.422467] Lustre: 297320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9093.484481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9093.488847] Lustre: Skipped 1 previous similar message [ 9093.566367] Lustre: lustre-MDT0001: new disk, initializing [ 9093.682091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9093.725011] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9093.741298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9098.087208] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9106.649639] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9112.621147] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9119.326390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9119.736543] Lustre: lustre-OST2710: new disk, initializing [ 9119.739996] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9119.745995] Lustre: 299290:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9119.832578] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9125.452510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9125.466337] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9125.612308] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9125.865882] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9134.667942] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9140.872304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9141.105514] Lustre: lustre-OST03e8: new disk, initializing [ 9141.108779] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9141.114675] Lustre: 300351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9141.207215] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9146.513590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9146.527135] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9146.597317] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9146.921353] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9153.746391] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9159.651684] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9159.789192] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9163.531874] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9163.754182] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9168.746616] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9169.044711] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9174.037054] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9174.189308] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9177.573319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9177.582654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9177.604682] Lustre: Skipped 3 previous similar messages [ 9177.631067] Lustre: Skipped 3 previous similar messages [ 9182.708676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9182.726831] Lustre: Skipped 3 previous similar messages [ 9183.710764] Lustre: server umount lustre-MDT0000 complete [ 9187.668670] LustreError: 297312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791927 with bad export cookie 16630149449579416521 [ 9187.685689] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9187.697146] LustreError: 297312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9187.821245] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9187.830152] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9187.835946] Lustre: Skipped 1 previous similar message [ 9188.497895] Lustre: server umount lustre-MDT0001 complete [ 9194.450081] Lustre: server umount lustre-OST2710 complete [ 9198.277279] Lustre: server umount lustre-OST03e8 complete [ 9218.273692] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 9230.165797] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9261.161924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9261.429386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9261.454393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9261.531547] Lustre: lustre-MDT0000: new disk, initializing [ 9261.623230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9261.642427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9265.799212] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9275.646301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.776874] Lustre: 305600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9275.830563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9275.836113] Lustre: Skipped 1 previous similar message [ 9275.911629] Lustre: lustre-MDT0001: new disk, initializing [ 9275.990467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9276.029266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9276.039458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9279.972179] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9288.542900] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9293.145275] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9299.066195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9299.316300] Lustre: lustre-OST0000: new disk, initializing [ 9299.318554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9299.324124] Lustre: 307566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9299.394910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9305.683381] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9308.231350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9308.236887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9308.324188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9314.730481] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9319.700076] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9319.919659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9324.971910] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9325.202731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9329.127686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9329.136388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9329.156501] Lustre: Skipped 1 previous similar message [ 9329.176456] Lustre: Skipped 2 previous similar messages [ 9333.523341] Lustre: server umount lustre-OST0000 complete [ 9337.319341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9337.335829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9337.358241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9343.466451] Lustre: server umount lustre-MDT0000 complete [ 9344.487093] LustreError: 305606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.509633] LustreError: 305606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9347.094444] LustreError: 305591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792086 with bad export cookie 16630149449579418201 [ 9347.098026] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9347.105198] LustreError: 305591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9347.404880] Lustre: server umount lustre-MDT0001 complete [ 9355.770413] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:54:53 (1787792093) [ 9357.442804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9359.557249] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:54:56 (1787792096) [ 9362.852581] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9387.603781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9387.863824] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9387.886192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9388.048658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9391.893787] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9399.476422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9399.556181] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9399.585281] Lustre: 311426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9404.132204] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9425.607088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9425.817905] Lustre: Denying initial registration attempt from nid 192.168.201.114@tcp, specified as failover [ 9425.832936] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9425.839443] LustreError: 312877:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9425.845259] LustreError: 312877:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9425.849252] LustreError: 312877:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9425.901293] Lustre: server umount lustre-OST0000 complete [ 9425.904169] LustreError: 312877:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9435.620886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9435.629929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9435.641987] Lustre: Skipped 1 previous similar message [ 9435.644949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9435.651030] Lustre: Skipped 2 previous similar messages [ 9439.694910] Lustre: server umount lustre-MDT0000 complete [ 9441.248849] LustreError: 311436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9441.260446] LustreError: 311436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9443.265211] LustreError: 311417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792183 with bad export cookie 16630149449579419083 [ 9443.274096] LustreError: 311417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9443.274392] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9443.711615] Lustre: server umount lustre-MDT0001 complete [ 9455.241695] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 9457.830842] Key type lgssc unregistered [ 9458.102690] LNet: 313999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9458.107147] LNetError: 313999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9458.116955] LNet: Removed LNI 192.168.201.114@tcp [ 9458.911420] Key type .llcrypt unregistered [ 9458.915346] Key type ._llcrypt unregistered [ 9468.358775] Key type ._llcrypt registered [ 9468.360754] Key type .llcrypt registered [ 9468.445392] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9469.181796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9469.262058] alg: No test for adler32 (adler32-zlib) [ 9470.217514] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 9470.356149] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 9471.969207] Key type lgssc registered [ 9473.001690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9519.497502] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9527.838731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9527.859694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9529.178897] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9529.219762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9529.412867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9534.173621] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9544.831118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9545.035752] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9545.113286] Lustre: 316764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9545.431413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9550.159425] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9557.152805] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9562.124301] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9582.254104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9582.457943] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9582.690899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9588.410867] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9595.268986] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9598.944737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9598.959743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9598.972328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9599.467201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9599.473514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.489581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9599.492022] Lustre: Skipped 1 previous similar message [ 9603.460875] Lustre: server umount lustre-OST0000 complete [ 9607.137989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9607.160786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.179964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9609.699481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9609.701058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9609.727936] Lustre: Skipped 1 previous similar message [ 9613.422053] Lustre: server umount lustre-MDT0000 complete [ 9616.856539] LustreError: 317505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792356 with bad export cookie 3881787485700975137 [ 9616.864242] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9616.867967] LustreError: 317505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9617.165768] Lustre: server umount lustre-MDT0001 complete [ 9627.848404] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 9630.519528] Key type lgssc unregistered [ 9630.805764] LNet: 320341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9630.810847] LNetError: 320341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9630.822539] LNet: Removed LNI 192.168.201.114@tcp [ 9631.702242] Key type .llcrypt unregistered [ 9631.706361] Key type ._llcrypt unregistered [ 9655.972308] Key type ._llcrypt registered [ 9655.973872] Key type .llcrypt registered [ 9656.083885] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9658.033804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9658.073110] alg: No test for adler32 (adler32-zlib) [ 9659.285243] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 9659.558407] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 9661.255227] Key type lgssc registered [ 9662.836470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9672.104545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9672.124144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9673.770844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9679.351326] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9683.952147] LustreError: 321761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9683.959235] LustreError: 321761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9687.530857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.969686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9692.658340] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9698.950912] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9704.127311] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9711.930500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9712.319553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9718.387601] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9725.633994] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9733.483393] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 9734.113511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9734.121979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9734.127892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9734.134343] Lustre: Skipped 2 previous similar messages [ 9734.139982] Lustre: Skipped 1 previous similar message [ 9738.601134] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 9738.613228] Lustre: Skipped 1 previous similar message [ 9739.764943] Lustre: server umount lustre-MDT0000 complete [ 9743.614128] LustreError: 321742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792483 with bad export cookie 5633500347632423350 [ 9743.622983] LustreError: 321742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9743.631789] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9743.703854] LustreError: 323668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9743.728960] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 9743.736548] Lustre: Skipped 3 previous similar messages [ 9744.357830] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.388202] LustreError: 323668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.388600] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9748.403127] LustreError: 323668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9748.414997] Lustre: Skipped 1 previous similar message [ 9749.930315] Lustre: server umount lustre-MDT0001 complete [ 9755.624171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9763.442491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9768.928043] LustreError: 321071:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d157c92e x1874636202616320/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9769.317207] LustreError: 323592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9769.337750] LustreError: 323592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9769.442980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9769.505668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9773.470847] LustreError: 325368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9775.216193] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9778.691969] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9783.778126] LustreError: 325366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9783.798901] LustreError: 325366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9783.885671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9784.298310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9789.230044] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9789.422890] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9796.058203] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9798.249727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9804.507746] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9806.141291] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9815.020452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9815.020622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9815.036354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9815.048138] Lustre: Skipped 2 previous similar messages [ 9825.248622] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9825.734252] Lustre: server umount lustre-OST0000 complete [ 9830.371468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.381048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9830.386284] Lustre: Skipped 2 previous similar messages [ 9835.489928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9835.507801] Lustre: Skipped 7 previous similar messages [ 9836.586407] Lustre: server umount lustre-MDT0000 complete [ 9840.456850] LustreError: 323605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792580 with bad export cookie 5633500347632431162 [ 9840.482156] LustreError: 323605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9840.487389] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9840.997575] Lustre: server umount lustre-MDT0001 complete [ 9852.123362] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 9855.389745] Key type lgssc unregistered [ 9855.750373] LNet: 328043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9855.758568] LNetError: 328043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9855.781862] LNet: Removed LNI 192.168.201.114@tcp [ 9856.983676] Key type .llcrypt unregistered [ 9856.991334] Key type ._llcrypt unregistered [ 9882.959885] Key type ._llcrypt registered [ 9882.968077] Key type .llcrypt registered [ 9883.121827] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9884.041251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9884.097151] alg: No test for adler32 (adler32-zlib) [ 9885.306610] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 9885.546144] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 9887.159475] Key type lgssc registered [ 9888.600226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9896.981404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9897.013231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.538491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9902.352272] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9908.716260] LustreError: 329460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9908.758859] LustreError: 329460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9912.831742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9913.511911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9918.201337] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9926.993585] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9932.738548] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9938.659086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9938.661802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9938.675064] Lustre: Skipped 1 previous similar message [ 9938.681527] Lustre: Skipped 1 previous similar message [ 9941.937635] Lustre: server umount lustre-MDT0000 complete [ 9944.035518] LustreError: 330188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.057706] LustreError: 330188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9945.338290] LustreError: 329442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792685 with bad export cookie 3428489194551835172 [ 9945.358715] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9945.735439] Lustre: server umount lustre-MDT0001 complete [ 9962.659865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.248954] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9963.315614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9963.579369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9969.058397] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9979.177264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.329705] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9979.342427] Lustre: 332049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9979.548506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9984.569616] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9992.442816] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9998.284926] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10008.106807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10008.255928] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10008.261428] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10008.264363] LustreError: 333936:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10008.267199] LustreError: 333936:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10008.269334] LustreError: 333936:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10008.361576] Lustre: server umount lustre-OST0000 complete [10008.367947] LustreError: 333936:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10018.611192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10018.799721] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10018.968966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10024.574697] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10025.063515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10030.958884] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10038.955340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10039.165461] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10039.231299] Lustre: lustre-OST0001: new disk, initializing [10039.233267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10039.236803] Lustre: 335477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10039.315923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10045.764478] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10048.222960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10048.242255] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10048.423097] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10054.476609] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10058.730464] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10058.733530] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10058.737560] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10058.750186] Lustre: Skipped 1 previous similar message [10063.848601] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10063.863257] Lustre: Skipped 2 previous similar messages [10064.051308] Lustre: server umount lustre-OST0001 complete [10068.963975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.968315] LustreError: 336280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10068.974508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10068.981069] Lustre: Skipped 1 previous similar message [10069.009214] LustreError: 336280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10074.093325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10074.095575] LustreError: 336280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10074.104430] Lustre: Skipped 2 previous similar messages [10074.133336] LustreError: 336280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10079.205547] LustreError: 336482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10079.230788] LustreError: 336482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10082.271569] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10082.580885] Lustre: server umount lustre-OST0000 complete [10087.401732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10087.427862] LustreError: Skipped 1 previous similar message [10087.438370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10087.475127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10087.485074] Lustre: Skipped 2 previous similar messages [10093.252162] Lustre: server umount lustre-MDT0000 complete [10094.560828] LustreError: 332061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10094.599392] LustreError: 332061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10098.094908] LustreError: 332041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792837 with bad export cookie 3428489194551835725 [10098.099918] LustreError: 332041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10098.107062] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10098.379705] Lustre: server umount lustre-MDT0001 complete [10111.391708] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [10115.184241] Key type lgssc unregistered [10115.712684] LNet: 337440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10115.722115] LNetError: 337440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10116.780242] LNet: Removed LNI 192.168.201.114@tcp [10118.017486] Key type .llcrypt unregistered [10118.019343] Key type ._llcrypt unregistered [10204.141535] Key type ._llcrypt registered [10204.144461] Key type .llcrypt registered [10204.248591] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [10217.356756] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [10218.307772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10218.331956] alg: No test for adler32 (adler32-zlib) [10219.344610] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [10219.574587] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [10221.327147] Key type lgssc registered [10222.783946] Lustre: Echo OBD driver; http://www.lustre.org/ [10265.675231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10265.748860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10267.118861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10267.171167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10267.259250] Lustre: lustre-MDT0000: new disk, initializing [10267.329500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10267.344410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10273.474115] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10287.167384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10287.286426] Lustre: 342360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10287.338539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10287.341171] Lustre: Skipped 1 previous similar message [10287.404137] Lustre: lustre-MDT0001: new disk, initializing [10287.493290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10287.542897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10287.554447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10291.489735] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10303.340720] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10309.231970] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10316.859939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10317.226803] Lustre: lustre-OST0000: new disk, initializing [10317.244512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10317.251653] Lustre: 344329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10317.423919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10325.290326] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10326.701862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10326.726213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10326.957634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10336.464553] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10341.461721] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10341.706762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10345.520022] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10345.717192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10352.612776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10352.612805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10352.630501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10352.650898] Lustre: Skipped 1 previous similar message [10354.584813] Lustre: server umount lustre-OST0000 complete [10359.268321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10359.277737] LustreError: Skipped 1 previous similar message [10359.283450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10359.305964] Lustre: Skipped 1 previous similar message [10359.314501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10362.851537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10362.854971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10365.431605] Lustre: server umount lustre-MDT0000 complete [10370.097726] LustreError: 342351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793109 with bad export cookie 4024993901895514350 [10370.104169] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10370.123029] LustreError: 342351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10370.476631] Lustre: server umount lustre-MDT0001 complete [10379.977362] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:11:56 (1787793116) [10389.375913] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:12:06 (1787793126) [10399.231433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10399.770146] LustreError: 347084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10399.781775] LustreError: 347084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10399.855426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10405.086488] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10405.344682] LustreError: 347085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10409.446111] LustreError: 347084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10414.567097] LustreError: 347085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.261972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10416.718134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10421.979254] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10430.731560] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10437.299556] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10444.965547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10445.221729] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [10445.229285] Lustre: Skipped 1 previous similar message [10445.359258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10452.976183] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10460.875822] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10468.395741] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10470.582942] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10472.680834] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10475.002420] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10477.267866] Lustre: Failing over lustre-MDT0000 [10477.713922] Lustre: server umount lustre-MDT0000 complete [10477.967385] LustreError: 347081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.050408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10478.060769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10482.154759] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10482.169744] Lustre: Skipped 1 previous similar message [10487.266499] LustreError: 348183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10487.282000] LustreError: 348183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10496.662480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10496.822387] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10497.130475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10497.217337] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10498.393050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10502.215511] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10502.640810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10502.679970] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10502.751558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10511.846647] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10513.542680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10517.431979] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10519.371486] Lustre: Failing over lustre-MDT0000 [10519.424659] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [10521.667636] Lustre: server umount lustre-MDT0000 complete [10523.109344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10523.114632] LustreError: 347079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10523.130497] Lustre: Skipped 1 previous similar message [10523.135549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10523.175893] LustreError: 347079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10538.465304] Lustre: 340582:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793262/real 1787793262] req@0000000089ea9e8e x1874636789867520/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787793278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10538.505194] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10540.720067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.725529] LustreError: 340578:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f5550784 x1874636789872896/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10549.040724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10549.103462] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10551.135409] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10554.158483] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10554.351948] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10554.362028] Lustre: Skipped 2 previous similar messages [10554.445735] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10554.535495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10564.935811] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10566.698725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10574.817083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10574.820625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10574.821710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10574.840355] Lustre: Skipped 3 previous similar messages [10574.868578] Lustre: Skipped 2 previous similar messages [10579.886673] Lustre: server umount lustre-MDT0000 complete [10579.937478] LustreError: 347084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.960968] LustreError: 347084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [10583.917741] LustreError: 347065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793323 with bad export cookie 4024993901895517094 [10583.938312] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10584.255404] Lustre: server umount lustre-MDT0001 complete [10598.708335] Lustre: server umount lustre-OST0000 complete [10617.709866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10618.309153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10623.816088] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10634.054918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10639.213336] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10648.039891] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10656.266893] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10664.171741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10672.958075] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10681.312453] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10688.434952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10698.721251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10698.726337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10698.743034] Lustre: Skipped 1 previous similar message [10705.028597] Lustre: server umount lustre-OST0000 complete [10711.014282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10716.826177] Lustre: server umount lustre-MDT0000 complete [10719.205893] LustreError: 353428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10719.216590] LustreError: 353428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10721.480326] LustreError: 354162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793461 with bad export cookie 4024993901895517808 [10721.497250] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10721.503322] LustreError: 354162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10722.099679] Lustre: server umount lustre-MDT0001 complete [10736.598901] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [10740.522258] Key type lgssc unregistered [10740.882336] LNet: 356985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10740.889254] LNetError: 356985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10740.901438] LNet: Removed LNI 192.168.201.114@tcp [10741.796759] Key type .llcrypt unregistered [10741.799233] Key type ._llcrypt unregistered [10774.301911] Key type ._llcrypt registered [10774.305190] Key type .llcrypt registered [10774.440133] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [10776.049429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10776.115810] alg: No test for adler32 (adler32-zlib) [10777.330378] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [10777.580985] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [10779.367220] Key type lgssc registered [10780.721136] Lustre: Echo OBD driver; http://www.lustre.org/ [10790.645921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10790.679793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.424579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10798.416572] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10802.673561] LustreError: 358503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.689692] LustreError: 358503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10807.777959] LustreError: 358502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.525055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.135885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10814.710387] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10823.410768] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10830.246330] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10839.509251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10839.901714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10848.217281] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10857.528683] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10862.462064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10875.872221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10875.882373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10875.901519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10877.934992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10877.950431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10877.968839] Lustre: Skipped 2 previous similar messages [10883.043792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10883.055333] Lustre: Skipped 2 previous similar messages [10888.166674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10890.207583] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10890.438757] Lustre: server umount lustre-MDT0000 complete [10893.280526] LustreError: 359230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.316139] LustreError: 359230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10894.544634] LustreError: 358483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793634 with bad export cookie 16724718418686341331 [10894.552441] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10894.555884] LustreError: 358483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10895.164799] Lustre: server umount lustre-MDT0001 complete [10910.285476] Lustre: server umount lustre-OST0000 complete [10924.033783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10924.673233] LustreError: 362241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10924.798254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10930.153419] LustreError: 362242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.423597] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10939.375262] LustreError: 362242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.397555] LustreError: 362242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10940.094521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10940.464519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10945.817841] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10955.021579] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10962.320445] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10970.210548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10970.649227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10978.709503] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10987.287476] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10995.269296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11012.069139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11012.077827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11012.095703] Lustre: Skipped 1 previous similar message [11012.107607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11012.118032] Lustre: Skipped 2 previous similar messages [11015.655754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11015.677371] Lustre: Skipped 1 previous similar message [11017.422559] Lustre: server umount lustre-MDT0000 complete [11020.775645] LustreError: 364915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.792952] LustreError: 364915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11021.516909] LustreError: 362223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793761 with bad export cookie 16724718418686342444 [11021.525865] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11021.539515] LustreError: 362223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11022.009927] Lustre: server umount lustre-MDT0001 complete [11036.980629] Lustre: server umount lustre-OST0000 complete [11055.644385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11056.201314] LustreError: 366175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.286086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11060.750621] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11069.726278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11075.226438] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11083.421905] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11089.336726] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11096.412062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11096.804376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11096.809892] Lustre: Skipped 1 previous similar message [11100.921276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11103.639989] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11111.055774] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11120.614894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11120.616485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11120.631981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11120.637535] Lustre: Skipped 1 previous similar message [11120.640116] Lustre: Skipped 3 previous similar messages [11122.835092] Lustre: server umount lustre-OST0000 complete [11130.848606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11134.159419] Lustre: server umount lustre-MDT0000 complete [11135.968374] LustreError: 366903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11135.978121] LustreError: 366903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11138.793938] LustreError: 366155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793878 with bad export cookie 16724718418686343466 [11138.800404] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11138.802845] LustreError: 366155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11139.489474] Lustre: server umount lustre-MDT0001 complete [11154.331634] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11157.865240] Key type lgssc unregistered [11158.343923] LNet: 369710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11158.347859] LNetError: 369710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11158.360257] LNet: Removed LNI 192.168.201.114@tcp [11159.566719] Key type .llcrypt unregistered [11159.570886] Key type ._llcrypt unregistered [11188.075355] Key type ._llcrypt registered [11188.080872] Key type .llcrypt registered [11188.239993] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11189.407211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11189.599609] alg: No test for adler32 (adler32-zlib) [11190.768271] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [11191.126031] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11192.903279] Key type lgssc registered [11194.338582] Lustre: Echo OBD driver; http://www.lustre.org/ [11203.397694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11203.422710] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.430772] LustreError: 371152:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11203.459085] LustreError: 371152:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11203.472458] LustreError: 371152:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11203.479784] LustreError: 371152:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11203.483546] LustreError: 371152:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11212.478142] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11212.487436] LustreError: 371551:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11212.507637] LustreError: 371551:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11212.520812] LustreError: 371551:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11212.528985] LustreError: 371551:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11212.539466] LustreError: 371551:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11233.396745] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11236.905973] Key type lgssc unregistered [11237.275549] LNet: 372557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11237.289881] LNetError: 372557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11237.339132] LNet: Removed LNI 192.168.201.114@tcp [11238.380225] Key type .llcrypt unregistered [11238.383567] Key type ._llcrypt unregistered [11258.875338] Key type ._llcrypt registered [11258.880657] Key type .llcrypt registered [11259.103634] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [11275.249516] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11276.317907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11276.386890] alg: No test for adler32 (adler32-zlib) [11277.556408] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [11277.904915] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11279.599306] Key type lgssc registered [11280.816309] Lustre: Echo OBD driver; http://www.lustre.org/ [11319.040282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11319.064170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11320.463732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11320.486171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11320.606739] Lustre: lustre-MDT0000: new disk, initializing [11320.716151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11320.735679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11326.161556] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11336.668314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11336.761736] Lustre: 375719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11336.817340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11336.820085] Lustre: Skipped 1 previous similar message [11336.894477] Lustre: lustre-MDT0001: new disk, initializing [11337.005696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11337.051696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11337.072924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11342.160250] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11351.876266] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11357.686900] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11364.117271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11364.353530] Lustre: lustre-OST0000: new disk, initializing [11364.357305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11364.364548] Lustre: 377684:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11364.462956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11370.638448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11370.648820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11370.865268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11371.269216] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11382.157393] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11387.509835] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11387.859685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11393.753363] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11394.005602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11396.581294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11396.588331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.596988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11396.635584] Lustre: Skipped 1 previous similar message [11401.724200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11401.734508] Lustre: Skipped 1 previous similar message [11402.783902] Lustre: server umount lustre-OST0000 complete [11408.875685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11408.890770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11408.923169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11408.933321] Lustre: Skipped 1 previous similar message [11411.950254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11411.977576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11414.452309] Lustre: server umount lustre-MDT0000 complete [11418.742655] LustreError: 376512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794158 with bad export cookie 3631355908197809598 [11418.751103] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11418.769629] LustreError: 376512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11419.199747] Lustre: server umount lustre-MDT0001 complete [11428.536158] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11429.815265] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:29:27 (1787794167) [11434.880680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11435.170070] LustreError: 380142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11435.181658] LustreError: 380142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11435.230472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11440.616292] LustreError: 380143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.657279] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11444.715613] LustreError: 380142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.707365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11449.825628] LustreError: 380143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11450.208239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11455.758655] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11464.881402] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11470.895946] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11478.566377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11478.888471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11485.976399] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11494.499088] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11501.295803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11501.548455] Lustre: lustre-OST0001: new disk, initializing [11501.556426] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11501.563359] Lustre: 382975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11501.642384] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11508.069435] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11508.837208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11508.841939] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11509.027030] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11517.157423] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11523.976441] Lustre: lustre-OST0001: Not available for connect from 192.168.201.14@tcp (stopping) [11523.988117] Lustre: Skipped 1 previous similar message [11524.595986] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11524.597376] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11524.623572] Lustre: Skipped 1 previous similar message [11527.224720] Lustre: server umount lustre-OST0001 complete [11529.048188] LustreError: 381965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.942253] LustreError: 381965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.971575] LustreError: 381965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11545.571352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11545.592522] LustreError: Skipped 1 previous similar message [11545.594475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11545.637582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11545.648840] Lustre: Skipped 2 previous similar messages [11551.788977] Lustre: server umount lustre-OST0000 complete [11557.857833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11557.866850] LustreError: Skipped 1 previous similar message [11557.869880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11557.881720] Lustre: Skipped 1 previous similar message [11563.781793] Lustre: server umount lustre-MDT0000 complete [11565.536931] LustreError: 380138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11565.554986] LustreError: 380138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11568.109709] LustreError: 380167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794307 with bad export cookie 3631355908197810480 [11568.119772] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11568.123154] LustreError: 380167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11574.499470] Lustre: server umount lustre-MDT0001 complete [11586.835608] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11590.241690] Key type lgssc unregistered [11590.497359] LNet: 384966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11590.504519] LNetError: 384966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11590.517803] LNet: Removed LNI 192.168.201.114@tcp [11591.417864] Key type .llcrypt unregistered [11591.420564] Key type ._llcrypt unregistered [11651.213277] Key type ._llcrypt registered [11651.215443] Key type .llcrypt registered [11651.312909] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11652.456349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11652.646108] alg: No test for adler32 (adler32-zlib) [11653.871917] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [11654.088104] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11655.770571] Key type lgssc registered [11657.330902] Lustre: Echo OBD driver; http://www.lustre.org/ [11667.384395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11667.408486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11668.740878] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11668.778191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11669.043980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11674.344991] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11684.870663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11684.974445] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11685.024948] Lustre: 387795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11685.488757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11691.313458] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11700.759897] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11707.383983] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11710.949950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11710.962032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.987292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11711.464074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11716.052905] Lustre: server umount lustre-MDT0000 complete [11720.057471] LustreError: 387786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794459 with bad export cookie 10812039814652544263 [11720.071028] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11720.072324] LustreError: 387786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11720.558736] Lustre: server umount lustre-MDT0001 complete [11731.537947] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:34:28 (1787794468) [11739.314393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.944746] LustreError: 390310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11739.971306] LustreError: 390310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11740.031270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11745.249018] LustreError: 390311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11745.275772] LustreError: 390311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11745.286301] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11750.375539] LustreError: 390310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11754.412039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11754.797559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11759.465404] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11767.732481] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11773.551549] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11781.224079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11781.374986] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11781.539796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11787.971849] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11796.102312] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11805.039254] Lustre: Permanently deactivating lustre-OST0000 [11805.057265] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11813.911669] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11816.922823] LustreError: 393345:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11816.934748] LustreError: 393345:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11816.944861] LustreError: 393345:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11820.331816] LustreError: 393394:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11820.336924] LustreError: 393394:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11829.167829] Lustre: server umount lustre-OST0000 complete [11831.990508] LustreError: 393643:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11831.997606] LustreError: 393643:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11835.363728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.396613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11835.404841] Lustre: Skipped 3 previous similar messages [11840.480853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11840.500576] Lustre: Skipped 1 previous similar message [11843.652386] Lustre: server umount lustre-MDT0000 complete [11845.600488] LustreError: 392152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.617551] LustreError: 392152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11847.183255] LustreError: 390290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794586 with bad export cookie 10812039814652544844 [11847.193447] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11847.195540] LustreError: 390290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11856.531599] Lustre: server umount lustre-MDT0001 complete [11865.801316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11870.580465] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11873.391704] LustreError: 394803:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11879.231623] Lustre: 395142:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.114@tcp [11881.274937] Lustre: server umount MGS complete [11891.138673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11891.508908] LustreError: 395866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11891.535333] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11891.542428] Lustre: Skipped 1 previous similar message [11891.586174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11896.408634] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11901.926309] LustreError: 395867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11901.963166] LustreError: 395867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11905.831982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11906.087465] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11910.721543] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11918.203440] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11924.165643] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11925.949196] Lustre: Permanently reactivating lustre-OST0000 [11925.980857] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11925.996785] Lustre: Skipped 2 previous similar messages [11955.175382] LustreError: 397697:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11965.157405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11965.637340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11965.640165] Lustre: Skipped 1 previous similar message [11971.911588] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11980.988443] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11986.401333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11986.409734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11986.419203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11986.454632] Lustre: Skipped 2 previous similar messages [11986.468891] Lustre: Skipped 1 previous similar message [11987.813397] Lustre: server umount lustre-OST0000 complete [11993.060587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11993.065206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11993.088127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12001.760692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12001.763441] Lustre: Skipped 2 previous similar messages [12006.367581] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12006.536236] Lustre: server umount lustre-MDT0000 complete [12006.884918] LustreError: 395862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12006.904576] LustreError: 395862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12010.307274] LustreError: 396602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794750 with bad export cookie 10812039814652545894 [12010.308831] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12010.317675] LustreError: 396602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12010.870615] Lustre: server umount lustre-MDT0001 complete [12022.831969] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12025.602814] Key type lgssc unregistered [12025.853570] LNet: 400284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12025.859300] LNetError: 400284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12026.919505] LNet: Removed LNI 192.168.201.114@tcp [12027.796547] Key type .llcrypt unregistered [12027.797963] Key type ._llcrypt unregistered [12046.502822] Key type ._llcrypt registered [12046.505569] Key type .llcrypt registered [12046.591770] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [12061.156604] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12062.385883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12062.441125] alg: No test for adler32 (adler32-zlib) [12063.537314] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [12063.784632] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [12065.519202] Key type lgssc registered [12066.369486] Lustre: Echo OBD driver; http://www.lustre.org/ [12101.852935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12101.900830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12103.286986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12103.320854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12103.408784] Lustre: lustre-MDT0000: new disk, initializing [12103.510826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12103.545343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12108.465148] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12120.249451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12120.419849] Lustre: 403444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12120.462287] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12120.466945] Lustre: Skipped 1 previous similar message [12120.527414] Lustre: lustre-MDT0001: new disk, initializing [12120.613966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12120.646122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12120.657923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12125.944394] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12135.207101] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12142.007331] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12149.715408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12149.919314] Lustre: lustre-OST0000: new disk, initializing [12149.922823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12149.926254] Lustre: 405412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12149.983145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12157.191702] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12159.556633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12159.572359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12159.677077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12167.058897] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12172.117854] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12172.337718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12176.896286] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12177.114423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12180.458736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12180.459628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12180.474619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12180.489169] Lustre: Skipped 1 previous similar message [12185.573479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12185.581171] Lustre: Skipped 1 previous similar message [12190.717863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12190.736711] Lustre: Skipped 2 previous similar messages [12193.247197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12193.397494] Lustre: server umount lustre-OST0000 complete [12200.936240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12200.953953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12200.962651] Lustre: Skipped 1 previous similar message [12200.974572] Lustre: Skipped 1 previous similar message [12206.049303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12206.063128] Lustre: Skipped 1 previous similar message [12212.191346] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12212.348732] LustreError: 406955:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000048f4d102) refcount nonzero (2) after lock cleanup; forcing cleanup. [12212.814676] Lustre: server umount lustre-MDT0000 complete [12217.094135] LustreError: 403437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794956 with bad export cookie 975232865050773679 [12217.099049] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12217.113899] LustreError: 403437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12217.619695] Lustre: server umount lustre-MDT0001 complete [12227.520742] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12229.416445] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:42:46 (1787794966) [12236.322234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12236.789266] LustreError: 407877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12236.806391] LustreError: 407877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12236.854478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12241.253764] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12241.892241] LustreError: 407878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.021425] LustreError: 407877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12249.730265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12250.077315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12254.456315] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12262.154125] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12268.211672] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12275.725672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12276.041936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12282.116386] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12289.508627] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12291.395824] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12294.056484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12304.864230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12304.875604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12304.877334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12304.877341] Lustre: Skipped 2 previous similar messages [12304.901793] LustreError: Skipped 1 previous similar message [12310.288490] Lustre: server umount lustre-OST0000 complete [12315.109070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12315.122187] Lustre: Skipped 2 previous similar messages [12319.908554] Lustre: server umount lustre-MDT0000 complete [12320.224577] LustreError: 407878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12320.244735] LustreError: 407878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12324.104022] LustreError: 407860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795063 with bad export cookie 975232865050774568 [12324.115962] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12324.516242] Lustre: server umount lustre-MDT0001 complete [12336.311572] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12340.579952] Key type lgssc unregistered [12341.249546] LNet: 411603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12341.288438] LNetError: 411603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12342.332522] LNet: Removed LNI 192.168.201.114@tcp [12343.838336] Key type .llcrypt unregistered [12343.842307] Key type ._llcrypt unregistered [12373.408519] Key type ._llcrypt registered [12373.421960] Key type .llcrypt registered [12373.585778] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12373.913863] Key type .llcrypt unregistered [12373.924189] Key type ._llcrypt unregistered [12391.778805] Key type ._llcrypt registered [12391.781740] Key type .llcrypt registered [12391.909447] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12392.728257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12392.737248] alg: No test for adler32 (adler32-zlib) [12394.093348] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [12394.380056] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [12396.119216] Key type lgssc registered [12397.399919] Lustre: Echo OBD driver; http://www.lustre.org/ [12406.573059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12406.590589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12407.908062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12412.859921] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12417.519795] LustreError: 413822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.545625] LustreError: 413822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12421.827450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12422.277054] LustreError: 413821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12422.332407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12427.749692] LustreError: 413822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12427.769886] LustreError: 413822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12429.736596] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12432.868517] LustreError: 414566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12432.894677] LustreError: 414566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12432.899690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12437.992957] LustreError: 413821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12438.020255] LustreError: 413821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12439.240201] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12446.424518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12447.000196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12452.550384] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12462.569288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12462.575240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12462.598799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12462.612281] Lustre: Skipped 1 previous similar message [12467.690602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12467.705178] Lustre: Skipped 1 previous similar message [12472.810936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12472.823018] Lustre: Skipped 1 previous similar message [12475.872050] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12476.090352] Lustre: server umount lustre-OST0000 complete [12483.057109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12483.057284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12483.060543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12483.077491] Lustre: Skipped 2 previous similar messages [12488.166029] LustreError: 413821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12488.187502] LustreError: 413821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12488.723156] Lustre: server umount lustre-MDT0000 complete [12492.636874] LustreError: 413802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795232 with bad export cookie 7398282281003395373 [12492.645938] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12493.250709] Lustre: server umount lustre-MDT0001 complete [12502.759206] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12506.769862] Key type lgssc unregistered [12506.999141] LNet: 416975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12507.019300] LNetError: 416975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12508.078935] LNet: Removed LNI 192.168.201.114@tcp [12509.539374] Key type .llcrypt unregistered [12509.546386] Key type ._llcrypt unregistered [12542.474553] Key type ._llcrypt registered [12542.475919] Key type .llcrypt registered [12542.553612] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12544.147356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12544.204480] alg: No test for adler32 (adler32-zlib) [12545.333606] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [12545.572044] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [12547.255705] Key type lgssc registered [12548.349962] Lustre: Echo OBD driver; http://www.lustre.org/ [12556.949201] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12556.978467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12572.578107] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12577.695333] LustreError: 418369:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.114@tcp: failed processing log, type 4: rc = -110 [12604.383221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12613.150721] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12623.042134] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12631.967138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12632.769753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12632.890554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12638.178727] LustreError: 418394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.187916] LustreError: 418394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12638.398462] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12639.234420] LustreError: 418395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.258926] LustreError: 418395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12644.339139] LustreError: 418394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.370234] LustreError: 418394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12648.181257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12648.775177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12653.669890] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12661.820127] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12668.061355] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12678.628040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12678.637412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12678.651923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12679.650685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12679.653922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12679.675546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12679.705023] Lustre: Skipped 1 previous similar message [12682.229326] Lustre: server umount lustre-OST0000 complete [12689.889439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12689.896222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12689.899794] Lustre: Skipped 1 previous similar message [12689.911052] Lustre: Skipped 1 previous similar message [12692.584402] Lustre: server umount lustre-MDT0000 complete [12695.011118] LustreError: 419409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12695.031594] LustreError: 419409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12697.040945] LustreError: 418377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795436 with bad export cookie 146892455979396276 [12697.057769] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12697.071729] LustreError: 418377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12697.698746] Lustre: server umount lustre-MDT0001 complete [12709.892868] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12713.462340] Key type lgssc unregistered [12713.712773] LNet: 421945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12713.717285] LNetError: 421945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12713.742265] LNet: Removed LNI 192.168.201.114@tcp [12714.719460] Key type .llcrypt unregistered [12714.721137] Key type ._llcrypt unregistered [12744.599476] Key type ._llcrypt registered [12744.604993] Key type .llcrypt registered [12744.753588] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12746.162463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12746.262290] alg: No test for adler32 (adler32-zlib) [12747.358714] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [12747.556752] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [12749.191440] Key type lgssc registered [12750.297717] Lustre: Echo OBD driver; http://www.lustre.org/ [12758.354543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12758.382961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12759.914486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12763.124306] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12769.257326] LustreError: 423364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12769.271097] LustreError: 423364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12771.154928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12771.683955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12775.807475] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12783.579637] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12789.658642] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12797.174141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12797.715897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12804.889321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12805.240465] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12813.836836] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12817.896257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12817.909926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12817.931938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12820.883495] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [12822.498163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12822.500332] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12824.320682] Lustre: server umount lustre-MDT0000 complete [12825.950617] LustreError: 423361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.317321] LustreError: 423343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.14@tcp arrived at 1787795567 with bad export cookie 10372378488365003893 [12827.336112] LustreError: 423343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12827.616832] LustreError: 423365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.659655] LustreError: 423365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12829.974888] LustreError: 423389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795569 with bad export cookie 10372378488365003004 [12829.995769] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12830.001546] LustreError: 423389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12832.736072] LustreError: 423360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12832.736740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12832.737423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12832.758489] LustreError: 423360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12832.792243] Lustre: Skipped 1 previous similar message [12832.797017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12832.809071] Lustre: Skipped 2 previous similar messages [12836.218344] Lustre: server umount lustre-OST0000 complete [12839.904805] LustreError: 423364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12839.937085] LustreError: 423364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12843.056626] Lustre: server umount lustre-MDT0001 complete [12854.527631] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12857.973710] Key type lgssc unregistered [12858.370674] LNet: 427050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12858.382733] LNetError: 427050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12858.419233] LNet: Removed LNI 192.168.201.114@tcp [12859.697618] Key type .llcrypt unregistered [12859.699363] Key type ._llcrypt unregistered [12886.508224] Key type ._llcrypt registered [12886.513237] Key type .llcrypt registered [12886.673367] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12887.841328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12888.129033] alg: No test for adler32 (adler32-zlib) [12889.173550] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [12889.420376] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [12891.079190] Key type lgssc registered [12892.432815] Lustre: Echo OBD driver; http://www.lustre.org/ [12900.254050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12900.281332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12901.718722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12906.035473] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12912.098160] LustreError: 428466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12912.108460] LustreError: 428466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12914.885297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12915.233893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12919.591795] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12927.763237] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12933.550929] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12941.069434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12941.527061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12946.677297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12948.070340] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12955.884429] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12961.252645] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12961.256331] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12961.263646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12963.674282] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [12965.346479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12965.350472] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12967.054596] Lustre: server umount lustre-MDT0001 complete [12970.467575] LustreError: 430208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12970.479928] LustreError: 430208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12972.512605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12972.518588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12972.539042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12972.541643] Lustre: Skipped 1 previous similar message [12975.585199] LustreError: 429194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12978.828744] Lustre: server umount lustre-OST0000 complete [12980.707399] LustreError: 429478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12983.680618] Lustre: server umount lustre-MDT0000 complete [12996.523595] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12998.896334] Key type lgssc unregistered [12999.128894] LNet: 432155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12999.134081] LNetError: 432155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12999.143485] LNet: Removed LNI 192.168.201.114@tcp [12999.895923] Key type .llcrypt unregistered [12999.899168] Key type ._llcrypt unregistered [13024.354481] Key type ._llcrypt registered [13024.357238] Key type .llcrypt registered [13024.471860] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [13039.459587] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [13041.000672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13041.056456] alg: No test for adler32 (adler32-zlib) [13042.139669] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [13042.425402] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [13044.095790] Key type lgssc registered [13045.135448] Lustre: Echo OBD driver; http://www.lustre.org/ [13082.087934] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [13100.528780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13100.596258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13102.092865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13102.134981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13102.356383] Lustre: lustre-MDT0000: new disk, initializing [13102.560915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13102.618953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13109.939722] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13125.955920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13126.088924] Lustre: lustre-OST0000: new disk, initializing [13126.090681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13126.092142] Lustre: Skipped 1 previous similar message [13126.094340] Lustre: 437193:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13126.152902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13131.938210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13131.946198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13131.984914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13132.791367] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13142.306534] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13157.472670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13157.627181] Lustre: 436171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13157.695506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13157.749660] Lustre: lustre-MDT0001: new disk, initializing [13157.831505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13157.865935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13157.870850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13162.259341] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13172.975504] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13173.218929] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13178.054582] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13178.366905] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13184.874204] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [13184.999370] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13185.002797] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13189.652940] Lustre: server umount lustre-MDT0001 complete [13195.232388] LustreError: 436179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13195.253330] LustreError: 436179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13200.377812] LustreError: 436184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13200.403742] LustreError: 436184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13205.483961] LustreError: 436185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13205.504980] LustreError: 436185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13209.056466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13209.064760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13209.074323] Lustre: Skipped 1 previous similar message [13209.077253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13209.081059] Lustre: Skipped 2 previous similar messages [13210.592981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13210.595699] LustreError: 436184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13215.712973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13215.713672] LustreError: 436185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.840550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13222.367531] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13222.530746] Lustre: server umount lustre-OST0000 complete [13225.963877] LustreError: 436184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13225.978578] LustreError: 436184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13233.094318] Lustre: server umount lustre-MDT0000 complete [13247.924966] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [13251.143488] Key type lgssc unregistered [13251.449581] LNet: 440943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13251.454657] LNetError: 440943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13251.466427] LNet: Removed LNI 192.168.201.114@tcp [13252.313643] Key type .llcrypt unregistered [13252.317246] Key type ._llcrypt unregistered [13294.512895] Key type ._llcrypt registered [13294.514514] Key type .llcrypt registered [13294.611518] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [13294.847749] Key type .llcrypt unregistered [13294.853527] Key type ._llcrypt unregistered [13314.381372] Key type ._llcrypt registered [13314.388464] Key type .llcrypt registered [13314.498223] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [13316.105542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13316.126539] alg: No test for adler32 (adler32-zlib) [13317.351679] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [13317.733816] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [13319.455271] Key type lgssc registered [13320.799965] Lustre: Echo OBD driver; http://www.lustre.org/ [13352.685355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13352.705743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13353.961174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13354.006673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13354.072712] Lustre: lustre-MDT0000: new disk, initializing [13354.144530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13354.159823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13358.562202] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13369.839068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13370.012969] Lustre: 444339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13370.100498] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13370.111703] Lustre: Skipped 1 previous similar message [13370.226508] Lustre: lustre-MDT0001: new disk, initializing [13370.373017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13370.437728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13370.448554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13376.056325] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13384.405434] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13390.163485] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13397.841365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13398.258342] Lustre: lustre-OST0000: new disk, initializing [13398.270440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13398.281188] Lustre: 446307:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13398.381276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13404.612554] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13406.352668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13406.366128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13406.596895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13415.884778] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13426.659614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13426.665080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13426.687443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13427.196862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13427.199494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13427.212634] Lustre: Skipped 1 previous similar message [13427.224913] Lustre: Skipped 2 previous similar messages [13429.952253] Lustre: server umount lustre-MDT0000 complete [13434.592069] LustreError: 444331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796174 with bad export cookie 4887947588061841170 [13434.606395] LustreError: 444331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13434.609533] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13434.923931] Lustre: server umount lustre-MDT0001 complete [13449.229341] Lustre: server umount lustre-OST0000 complete [13460.740491] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:03:17 (1787796197) [13468.630837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13469.165528] LustreError: 448336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13469.182479] LustreError: 448336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13469.235243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13474.274169] LustreError: 448337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13474.323548] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13478.368827] LustreError: 448336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13483.488434] LustreError: 448337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13484.124225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13484.520756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13489.419719] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13498.509095] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13505.076505] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13524.860612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13525.298176] Lustre: Found index 0 for lustre-OST0000, updating log [13525.514145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13533.073777] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [13540.015492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13542.265805] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13550.560582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13550.563214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13550.571585] LustreError: Skipped 1 previous similar message [13550.572372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13550.588831] Lustre: Skipped 1 previous similar message [13554.417078] Lustre: server umount lustre-OST0000 complete [13570.971184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13570.975327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13570.979703] Lustre: Skipped 1 previous similar message [13570.990654] Lustre: Skipped 4 previous similar messages [13576.163744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13576.172405] Lustre: Skipped 1 previous similar message [13584.354781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13584.367926] Lustre: Skipped 5 previous similar messages [13585.375254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13585.772611] Lustre: server umount lustre-MDT0000 complete [13589.475378] LustreError: 449327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13589.500826] LustreError: 449327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13589.828290] LustreError: 448317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796329 with bad export cookie 4887947588061842563 [13589.837534] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13589.842624] LustreError: 448317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13590.141961] Lustre: server umount lustre-MDT0001 complete [13593.311231] LNet: 1 local NIs in recovery (showing 1): 192.168.201.114@tcp [13599.224135] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:05:36 (1787796336) [14222.493467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14223.087078] LustreError: 454751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14223.240942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14228.029915] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [14228.452370] LustreError: 454752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14232.546623] LustreError: 454751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14236.870245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14237.356418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14242.147252] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [14249.012933] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14255.193886] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14274.676876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14274.901861] Lustre: Found index 0 for lustre-OST0000, updating log [14275.142659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14281.182509] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [14284.516228] LNet: 1 local NIs in recovery (showing 1): 192.168.201.114@tcp [14288.235869] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14290.433288] Lustre: server umount lustre-OST0000 complete [14293.984459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14293.990601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14294.000055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14295.015981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14296.033984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14296.042505] Lustre: Skipped 3 previous similar messages [14300.128755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14300.137211] Lustre: Skipped 7 previous similar messages [14308.319312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14308.344590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14308.352371] Lustre: Skipped 14 previous similar messages [14309.414403] LustreError: 455770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14309.433099] LustreError: 455770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14310.637448] Lustre: server umount lustre-MDT0000 complete [14314.606365] LustreError: 454733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797054 with bad export cookie 4887947588061843648 [14314.618906] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14314.625817] LustreError: 454733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14315.039396] Lustre: server umount lustre-MDT0001 complete [14345.445514] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [14355.040292] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [14399.853292] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:18:57 (1787797137) [14420.013423] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [14423.801201] Key type lgssc unregistered [14424.145837] LNet: 461367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14424.165499] LNetError: 461367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14537.829960] LNet: Removed LNI 192.168.201.114@tcp [14538.611367] Key type .llcrypt unregistered [14538.612675] Key type ._llcrypt unregistered [14555.408269] Key type ._llcrypt registered [14555.410886] Key type .llcrypt registered [14555.524772] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [14556.404456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14556.428227] alg: No test for adler32 (adler32-zlib) [14557.562657] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [14557.817772] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [14559.424059] Key type lgssc registered [14560.577857] Lustre: Echo OBD driver; http://www.lustre.org/ [14564.628641] LNet: Added LNI 192.168.201.114@tcp100 [8/256/0/180] [14565.761774] LNet: Added LNI 192.168.201.114@tcp101 [8/256/0/180] [14566.699812] LNet: Added LNI 192.168.201.114@tcp102 [8/256/0/180] [14567.534840] LNet: Added LNI 192.168.201.114@tcp103 [8/256/0/180] [14568.265662] LNet: Added LNI 192.168.201.114@tcp104 [8/256/0/180] [14568.994831] LNet: Added LNI 192.168.201.114@tcp105 [8/256/0/180] [14569.622296] LNet: Added LNI 192.168.201.114@tcp106 [8/256/0/180] [14570.607910] LNet: Added LNI 192.168.201.114@tcp107 [8/256/0/180] [14571.554982] LNet: Added LNI 192.168.201.114@tcp108 [8/256/0/180] [14572.269687] LNet: Added LNI 192.168.201.114@tcp109 [8/256/0/180] [14572.999656] LNet: Added LNI 192.168.201.114@tcp110 [8/256/0/180] [14573.754043] LNet: Added LNI 192.168.201.114@tcp111 [8/256/0/180] [14574.712746] LNet: Added LNI 192.168.201.114@tcp112 [8/256/0/180] [14575.690557] LNet: Added LNI 192.168.201.114@tcp113 [8/256/0/180] [14576.732431] LNet: Added LNI 192.168.201.114@tcp114 [8/256/0/180] [14577.970954] LNet: Added LNI 192.168.201.114@tcp115 [8/256/0/180] [14579.148797] LNet: Added LNI 192.168.201.114@tcp116 [8/256/0/180] [14580.115171] LNet: Added LNI 192.168.201.114@tcp117 [8/256/0/180] [14581.183082] LNet: Added LNI 192.168.201.114@tcp118 [8/256/0/180] [14582.303971] LNet: Added LNI 192.168.201.114@tcp119 [8/256/0/180] [14583.350521] LNet: Added LNI 192.168.201.114@tcp120 [8/256/0/180] [14584.377019] LNet: Added LNI 192.168.201.114@tcp121 [8/256/0/180] [14585.322805] LNet: Added LNI 192.168.201.114@tcp122 [8/256/0/180] [14586.130824] LNet: Added LNI 192.168.201.114@tcp123 [8/256/0/180] [14586.957531] LNet: Added LNI 192.168.201.114@tcp124 [8/256/0/180] [14587.870759] LNet: Added LNI 192.168.201.114@tcp125 [8/256/0/180] [14588.842664] LNet: Added LNI 192.168.201.114@tcp126 [8/256/0/180] [14589.921398] LNet: Added LNI 192.168.201.114@tcp127 [8/256/0/180] [14591.009079] LNet: Added LNI 192.168.201.114@tcp128 [8/256/0/180] [14592.210441] LNet: Added LNI 192.168.201.114@tcp129 [8/256/0/180] [14593.164736] LNet: Added LNI 192.168.201.114@tcp130 [8/256/0/180] [14594.051272] LNet: Added LNI 192.168.201.114@tcp131 [8/256/0/180] [14595.024732] LNet: Added LNI 192.168.201.114@tcp132 [8/256/0/180] [14596.085914] LNet: Added LNI 192.168.201.114@tcp133 [8/256/0/180] [14596.868056] LNet: Added LNI 192.168.201.114@tcp134 [8/256/0/180] [14597.798451] LNet: Added LNI 192.168.201.114@tcp135 [8/256/0/180] [14604.185779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14604.204441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14605.295207] Lustre: 464399:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [14605.357564] LNetError: 464430:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [14605.367195] LNetError: Protocol error connecting to 192.168.201.114@tcp116 at host 192.168.201.114:988: is it running a compatible version of Lustre? [14605.375577] LNetError: Skipped 8 previous similar messages [14605.407537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14605.430776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14605.503680] Lustre: lustre-MDT0000: new disk, initializing [14605.583101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14605.593525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14605.983104] LNet: 5 local NIs in recovery (showing 5): 192.168.201.114@tcp125, 192.168.201.114@tcp126, 192.168.201.114@tcp127, 192.168.201.114@tcp128, 192.168.201.114@tcp129 [14606.432122] LNetError: 464431:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [14606.436533] LNetError: 464431:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14606.440159] LNetError: Protocol error connecting to 192.168.201.114@tcp114 at host 192.168.201.114:988: is it running a compatible version of Lustre? [14606.444349] LNetError: Skipped 2 previous similar messages [14609.064454] LNetError: 464420:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [14609.067881] LNetError: 464420:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14609.075678] LNetError: Protocol error connecting to 192.168.201.114@tcp125 at host 192.168.201.114:988: is it running a compatible version of Lustre? [14609.084297] LNetError: Skipped 8 previous similar messages [14610.114612] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [14613.156409] LNetError: 461857:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [14613.169397] LNetError: 461857:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14613.192138] LNetError: Protocol error connecting to 192.168.201.114@tcp125 at host 192.168.201.114:988: is it running a compatible version of Lustre? [14621.350400] LNetError: 464421:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [14621.371187] LNetError: Protocol error connecting to 192.168.201.114@tcp125 at host 192.168.201.114:988: is it running a compatible version of Lustre? [14622.492441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14622.715223] Lustre: 464414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14622.795660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14622.804887] Lustre: Skipped 1 previous similar message [14622.937802] Lustre: lustre-MDT0001: new disk, initializing [14622.989452] Lustre: 465219:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [14622.996840] Lustre: 465219:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14623.096836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14623.131822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14623.144619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14628.652953] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [14630.444552] Lustre: 465562:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [14630.449212] Lustre: 465562:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14635.688582] LNetError: 461858:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [14635.710453] LNetError: Protocol error connecting to 192.168.201.114@tcp121 at host 192.168.201.114:988: is it running a compatible version of Lustre? [14639.232124] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14645.622971] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14652.066064] LNetError: 464435:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [14652.087872] LNetError: 464435:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [14652.098244] LNetError: Protocol error connecting to 192.168.201.114@tcp114 at host 192.168.201.114:988: is it running a compatible version of Lustre? [14652.111556] LNetError: Skipped 12 previous similar messages [14652.863918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14653.350479] Lustre: lustre-OST0000: new disk, initializing [14653.354372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14653.364541] Lustre: 466393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14653.458196] Lustre: 466398:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [14653.472395] Lustre: 466398:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14653.521958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14660.504337] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [14660.719205] Lustre: 466751:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [14660.734087] Lustre: 466751:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14660.846096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14660.861149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14661.013088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14665.439198] LNet: 1 local NIs in recovery (showing 1): 192.168.201.114@tcp125 [14670.003372] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14681.573635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14681.575489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14681.590215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14686.692566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14686.701766] Lustre: Skipped 1 previous similar message [14688.928918] LNetError: 461860:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [14688.942659] LNetError: 461860:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 19 previous similar messages [14688.956551] LNetError: Protocol error connecting to 192.168.201.114@tcp129 at host 192.168.201.114:988: is it running a compatible version of Lustre? [14688.967703] LNetError: Skipped 19 previous similar messages [14691.295124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14691.503456] Lustre: server umount lustre-OST0000 complete [14694.885909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14694.897235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14694.909170] Lustre: Skipped 1 previous similar message [14694.915780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14694.917354] Lustre: Skipped 1 previous similar message [14696.928716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14696.931561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14696.949857] Lustre: Skipped 1 previous similar message [14701.025156] Lustre: server umount lustre-MDT0000 complete [14704.990681] LustreError: 464409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797444 with bad export cookie 5023470740285009731 [14704.996976] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14705.001366] LustreError: 464409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14705.389456] Lustre: server umount lustre-MDT0001 complete [14716.434683] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [14719.597715] Key type lgssc unregistered [14720.204800] LNet: 468309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14720.213846] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14720.229361] LNet: Removed LNI 192.168.201.114@tcp [14721.311412] LNet: Removed LNI 192.168.201.114@tcp100 [14721.313594] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14721.329845] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14721.340777] LNet: Removed LNI 192.168.201.114@tcp101 [14722.399619] LNet: Removed LNI 192.168.201.114@tcp102 [14722.403222] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14722.415148] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14722.424358] LNet: Removed LNI 192.168.201.114@tcp103 [14723.488581] LNet: Removed LNI 192.168.201.114@tcp104 [14724.511587] LNet: Removed LNI 192.168.201.114@tcp105 [14724.532155] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14724.546649] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14724.559650] LNet: Removed LNI 192.168.201.114@tcp106 [14725.599346] LNet: Removed LNI 192.168.201.114@tcp107 [14726.623302] LNet: Removed LNI 192.168.201.114@tcp108 [14727.647816] LNet: Removed LNI 192.168.201.114@tcp109 [14728.671779] LNet: Removed LNI 192.168.201.114@tcp110 [14728.679867] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14728.691082] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14728.706062] LNet: Removed LNI 192.168.201.114@tcp111 [14729.762369] LNet: Removed LNI 192.168.201.114@tcp112 [14730.783319] LNet: Removed LNI 192.168.201.114@tcp113 [14731.807381] LNet: Removed LNI 192.168.201.114@tcp114 [14732.831217] LNet: Removed LNI 192.168.201.114@tcp115 [14733.855203] LNet: Removed LNI 192.168.201.114@tcp116 [14734.943385] LNet: Removed LNI 192.168.201.114@tcp117 [14735.967848] LNet: Removed LNI 192.168.201.114@tcp118 [14736.992160] LNet: Removed LNI 192.168.201.114@tcp119 [14736.994343] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14737.002484] LNetError: 468309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14737.013183] LNet: Removed LNI 192.168.201.114@tcp120 [14738.079131] LNet: Removed LNI 192.168.201.114@tcp121 [14739.103476] LNet: Removed LNI 192.168.201.114@tcp122 [14740.127145] LNet: Removed LNI 192.168.201.114@tcp123 [14741.151246] LNet: Removed LNI 192.168.201.114@tcp124 [14741.157672] LNet: Removed LNI 192.168.201.114@tcp125 [14742.175159] LNet: Removed LNI 192.168.201.114@tcp126 [14743.199437] LNet: Removed LNI 192.168.201.114@tcp127 [14744.233080] LNet: Removed LNI 192.168.201.114@tcp128 [14745.248796] LNet: Removed LNI 192.168.201.114@tcp129 [14745.254767] LNet: Removed LNI 192.168.201.114@tcp130 [14745.259651] LNet: Removed LNI 192.168.201.114@tcp131 [14745.267423] LNet: Removed LNI 192.168.201.114@tcp132 [14745.271315] LNet: Removed LNI 192.168.201.114@tcp133 [14745.278276] LNet: Removed LNI 192.168.201.114@tcp134 [14745.299816] LNet: Removed LNI 192.168.201.114@tcp135 [14746.262110] Key type .llcrypt unregistered [14746.263702] Key type ._llcrypt unregistered [14771.160404] Key type ._llcrypt registered [14771.167165] Key type .llcrypt registered [14771.267845] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [14785.707651] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [14787.005900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14787.218395] alg: No test for adler32 (adler32-zlib) [14788.343343] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [14788.695833] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [14790.391241] Key type lgssc registered [14791.543802] Lustre: Echo OBD driver; http://www.lustre.org/ [14829.697210] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:26:06 (1787797566) [14851.300225] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [14854.350966] Key type lgssc unregistered [14854.862286] LNet: 472400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14854.871110] LNetError: 472400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14854.905430] LNet: Removed LNI 192.168.201.114@tcp [14856.113593] Key type .llcrypt unregistered [14856.116289] Key type ._llcrypt unregistered [14876.639749] Key type ._llcrypt registered [14876.644450] Key type .llcrypt registered [14876.885664] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [14878.329734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14878.378111] alg: No test for adler32 (adler32-zlib) [14879.370736] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [14879.545970] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [14881.199156] Key type lgssc registered [14882.780384] Lustre: Echo OBD driver; http://www.lustre.org/ [14893.394973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14893.437351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14894.790608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14894.827910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14895.048483] Lustre: lustre-MDT0000: new disk, initializing [14895.178296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14895.222948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14900.380147] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [14911.280724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14911.437853] Lustre: 473617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14911.526220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14911.531284] Lustre: Skipped 1 previous similar message [14911.656920] Lustre: lustre-MDT0001: new disk, initializing [14911.807206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14911.851656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14911.866584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14916.616395] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [14926.235254] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14931.526339] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14938.867404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14939.222843] Lustre: lustre-OST0000: new disk, initializing [14939.226497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14939.231736] Lustre: 475585:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14939.307358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14945.371175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14945.377614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14945.543659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14946.501417] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [14956.978220] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14961.411941] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing lnet_if_list [14967.441172] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14967.479903] LNet: Added LNI 192.168.201.114@tcp10 [8/256/0/180] [14974.139926] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14974.194409] LNet: Added LNI 192.168.201.114@tcp11 [8/256/0/180] [14978.067301] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14978.088363] LNet: Added LNI 192.168.201.114@tcp12 [8/256/0/180] [14982.959867] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14983.006773] LNet: Added LNI 192.168.201.114@tcp13 [8/256/0/180] [14988.551273] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14988.605685] LNet: Added LNI 192.168.201.114@tcp14 [8/256/0/180] [14995.665722] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14995.707496] LNet: Added LNI 192.168.201.114@tcp15 [8/256/0/180] [15000.180759] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15000.225809] LNet: Added LNI 192.168.201.114@tcp16 [8/256/0/180] [15007.425839] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15007.456390] LNet: Added LNI 192.168.201.114@tcp17 [8/256/0/180] [15013.197759] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15013.229341] LNet: Added LNI 192.168.201.114@tcp18 [8/256/0/180] [15019.713416] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15019.755788] LNet: Added LNI 192.168.201.114@tcp19 [8/256/0/180] [15025.857219] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15025.885859] LNet: Added LNI 192.168.201.114@tcp20 [8/256/0/180] [15031.553885] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15031.575953] LNet: Added LNI 192.168.201.114@tcp21 [8/256/0/180] [15038.002545] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15038.026828] LNet: Added LNI 192.168.201.114@tcp22 [8/256/0/180] [15043.702483] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15043.724739] LNet: Added LNI 192.168.201.114@tcp23 [8/256/0/180] [15050.266217] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15050.308458] LNet: Added LNI 192.168.201.114@tcp24 [8/256/0/180] [15056.958602] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15056.980520] LNet: Added LNI 192.168.201.114@tcp25 [8/256/0/180] [15063.354331] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15063.385255] LNet: Added LNI 192.168.201.114@tcp26 [8/256/0/180] [15070.335553] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15070.368072] LNet: Added LNI 192.168.201.114@tcp27 [8/256/0/180] [15076.662303] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15076.690852] LNet: Added LNI 192.168.201.114@tcp28 [8/256/0/180] [15082.559972] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15082.588077] LNet: Added LNI 192.168.201.114@tcp29 [8/256/0/180] [15088.524090] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15088.550730] LNet: Added LNI 192.168.201.114@tcp30 [8/256/0/180] [15096.164216] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15096.200856] LNet: Added LNI 192.168.201.114@tcp31 [8/256/0/180] [15103.225709] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15103.255519] LNet: Added LNI 192.168.201.114@tcp32 [8/256/0/180] [15110.219667] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15110.254771] LNet: Added LNI 192.168.201.114@tcp33 [8/256/0/180] [15118.164924] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15118.214073] LNet: Added LNI 192.168.201.114@tcp34 [8/256/0/180] [15125.942158] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15125.999335] LNet: Added LNI 192.168.201.114@tcp35 [8/256/0/180] [15133.845048] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15133.892843] LNet: Added LNI 192.168.201.114@tcp36 [8/256/0/180] [15141.176346] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15141.208507] LNet: Added LNI 192.168.201.114@tcp37 [8/256/0/180] [15148.142936] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15148.177724] LNet: Added LNI 192.168.201.114@tcp38 [8/256/0/180] [15156.762114] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15156.792158] LNet: Added LNI 192.168.201.114@tcp39 [8/256/0/180] [15165.256713] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15165.295750] LNet: Added LNI 192.168.201.114@tcp40 [8/256/0/180] [15172.957392] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15172.994910] LNet: Added LNI 192.168.201.114@tcp41 [8/256/0/180] [15181.049913] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15181.093785] LNet: Added LNI 192.168.201.114@tcp42 [8/256/0/180] [15189.807689] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15189.852339] LNet: Added LNI 192.168.201.114@tcp43 [8/256/0/180] [15196.450436] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15196.483533] LNet: Added LNI 192.168.201.114@tcp44 [8/256/0/180] [15203.754554] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15203.773495] LNet: Added LNI 192.168.201.114@tcp45 [8/256/0/180] [15211.332168] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15211.373510] LNet: Added LNI 192.168.201.114@tcp46 [8/256/0/180] [15219.492957] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15219.523323] LNet: Added LNI 192.168.201.114@tcp47 [8/256/0/180] [15227.324221] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15227.348567] LNet: Added LNI 192.168.201.114@tcp48 [8/256/0/180] [15234.776512] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15234.817284] LNet: Added LNI 192.168.201.114@tcp49 [8/256/0/180] [15243.696228] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15243.737847] LNet: Added LNI 192.168.201.114@tcp50 [8/256/0/180] [15258.086881] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15258.102569] Lustre: Skipped 1 previous similar message [15258.117192] Lustre: server umount lustre-MDT0000 complete [15262.517515] LustreError: 473608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798002 with bad export cookie 7840270234722420244 [15262.541013] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15262.922553] Lustre: server umount lustre-MDT0001 complete [15278.559122] Lustre: 472887:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787798002/real 1787798002] req@0000000095de117c x1874641676315264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787798018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15278.588991] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15278.618313] Lustre: Skipped 1 previous similar message [15280.170262] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [15282.720779] Lustre: 472887:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787798006/real 1787798006] req@00000000a50f824c x1874641676315392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787798022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15284.704478] Lustre: 472889:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787798008/real 1787798008] req@000000007c669a66 x1874641676315520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787798024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15288.863713] Lustre: 472889:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787798012/real 1787798012] req@00000000dabd926b x1874641676315776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787798028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15289.397858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15293.921554] Lustre: 472887:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787798017/real 1787798017] req@0000000038979e4f x1874641676316032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787798033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15307.602026] LustreError: 475579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15307.622415] LustreError: 475579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15307.693693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15309.733046] LustreError: 475580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15312.013252] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15314.930386] LustreError: 475579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15320.035465] LustreError: 475580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15320.158537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15320.408829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15322.477396] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15323.502525] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15325.242640] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15331.801776] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15337.769316] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15356.402032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15356.419359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15356.439926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15358.947864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15358.956341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15358.972176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15362.637540] Lustre: server umount lustre-OST0000 complete [15369.712790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15369.721824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15369.742112] Lustre: Skipped 1 previous similar message [15369.756081] Lustre: Skipped 2 previous similar messages [15373.147470] Lustre: server umount lustre-MDT0000 complete [15374.821356] LustreError: 499830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15374.852019] LustreError: 499830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15377.379556] LustreError: 473607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798117 with bad export cookie 7840270234722422904 [15377.384107] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15377.401795] LustreError: 473607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [15377.765691] Lustre: server umount lustre-MDT0001 complete [15388.318891] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [15391.238775] Key type lgssc unregistered [15391.607851] LNet: 502631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15391.616632] LNetError: 502631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15392.673819] LNet: Removed LNI 192.168.201.114@tcp [15392.681392] LNetError: 502631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15392.693374] LNet: Removed LNI 192.168.201.114@tcp10 [15392.701488] LNet: Removed LNI 192.168.201.114@tcp11 [15392.706809] LNet: Removed LNI 192.168.201.114@tcp12 [15392.711373] LNet: Removed LNI 192.168.201.114@tcp13 [15392.716594] LNet: Removed LNI 192.168.201.114@tcp14 [15392.721979] LNet: Removed LNI 192.168.201.114@tcp15 [15392.731633] LNet: Removed LNI 192.168.201.114@tcp16 [15392.737775] LNet: Removed LNI 192.168.201.114@tcp17 [15392.747760] LNet: Removed LNI 192.168.201.114@tcp18 [15392.753096] LNet: Removed LNI 192.168.201.114@tcp19 [15392.757154] LNet: Removed LNI 192.168.201.114@tcp20 [15392.760223] LNet: Removed LNI 192.168.201.114@tcp21 [15392.767423] LNet: Removed LNI 192.168.201.114@tcp22 [15392.777718] LNet: Removed LNI 192.168.201.114@tcp23 [15392.779551] LNet: Removed LNI 192.168.201.114@tcp24 [15392.790927] LNet: Removed LNI 192.168.201.114@tcp25 [15392.804504] LNet: Removed LNI 192.168.201.114@tcp26 [15392.811652] LNet: Removed LNI 192.168.201.114@tcp27 [15392.820785] LNet: Removed LNI 192.168.201.114@tcp28 [15392.825181] LNet: Removed LNI 192.168.201.114@tcp29 [15392.826975] LNet: Removed LNI 192.168.201.114@tcp30 [15392.829489] LNet: Removed LNI 192.168.201.114@tcp31 [15392.837225] LNet: Removed LNI 192.168.201.114@tcp32 [15392.842788] LNet: Removed LNI 192.168.201.114@tcp33 [15392.855497] LNet: Removed LNI 192.168.201.114@tcp34 [15392.861821] LNet: Removed LNI 192.168.201.114@tcp35 [15392.874678] LNet: Removed LNI 192.168.201.114@tcp36 [15392.879539] LNet: Removed LNI 192.168.201.114@tcp37 [15392.885605] LNet: Removed LNI 192.168.201.114@tcp38 [15392.894628] LNet: Removed LNI 192.168.201.114@tcp39 [15392.908650] LNet: Removed LNI 192.168.201.114@tcp40 [15392.910366] LNet: Removed LNI 192.168.201.114@tcp41 [15392.929123] LNet: Removed LNI 192.168.201.114@tcp42 [15392.937883] LNet: Removed LNI 192.168.201.114@tcp43 [15392.941363] LNet: Removed LNI 192.168.201.114@tcp44 [15392.944767] LNet: Removed LNI 192.168.201.114@tcp45 [15392.948725] LNet: Removed LNI 192.168.201.114@tcp46 [15392.951733] LNet: Removed LNI 192.168.201.114@tcp47 [15392.959420] LNet: Removed LNI 192.168.201.114@tcp48 [15392.967750] LNet: Removed LNI 192.168.201.114@tcp49 [15392.977131] LNet: Removed LNI 192.168.201.114@tcp50 [15393.892375] Key type .llcrypt unregistered [15393.894313] Key type ._llcrypt unregistered [15415.872458] Key type ._llcrypt registered [15415.873841] Key type .llcrypt registered [15416.011515] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [15430.596085] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [15431.574588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15431.693672] alg: No test for adler32 (adler32-zlib) [15432.715933] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [15433.057615] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [15434.751656] Key type lgssc registered [15435.813857] Lustre: Echo OBD driver; http://www.lustre.org/ [15470.279392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15470.316581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15471.735753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15471.775589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15471.875746] Lustre: lustre-MDT0000: new disk, initializing [15472.010748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15472.034892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15477.006678] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15487.737098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15487.850707] Lustre: 505826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15487.909775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15487.913181] Lustre: Skipped 1 previous similar message [15487.997254] Lustre: lustre-MDT0001: new disk, initializing [15488.183376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15488.240571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15488.258528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15493.436711] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15502.074394] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15507.983220] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15516.199123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15516.390834] Lustre: *** cfs_fail_loc=90f, val=0*** [15516.470260] Lustre: lustre-OST0000: new disk, initializing [15516.476066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15516.482960] Lustre: 507844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15516.580025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15522.311290] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15524.459427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15524.471951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15524.614001] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15532.073644] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15540.195191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15540.205328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15540.214226] Lustre: Skipped 1 previous similar message [15540.239124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15545.319516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15545.325799] Lustre: Skipped 2 previous similar messages [15545.913496] Lustre: server umount lustre-OST0000 complete [15549.928151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15549.947650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15549.979929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15555.552715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15555.558362] Lustre: Skipped 3 previous similar messages [15555.962798] Lustre: server umount lustre-MDT0000 complete [15559.671253] LustreError: 505819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798299 with bad export cookie 252651813134704694 [15559.680318] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15559.693284] LustreError: 505819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15560.300677] Lustre: server umount lustre-MDT0001 complete [15571.217669] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [15573.913572] Key type lgssc unregistered [15574.213945] LNet: 509752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15574.220139] LNetError: 509752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15574.234809] LNet: Removed LNI 192.168.201.114@tcp [15575.141424] Key type .llcrypt unregistered [15575.144887] Key type ._llcrypt unregistered [15594.259800] Key type ._llcrypt registered [15594.263825] Key type .llcrypt registered [15594.372097] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [15595.947270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15595.960215] alg: No test for adler32 (adler32-zlib) [15597.015952] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [15597.272769] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [15598.959407] Key type lgssc registered [15600.043268] Lustre: Echo OBD driver; http://www.lustre.org/ [15607.347193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15607.365678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15608.870380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15614.174764] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15619.042819] LustreError: 511023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15619.074183] LustreError: 511023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15622.749609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15623.150881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15627.842349] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15635.468396] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15641.452845] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15648.138286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15648.448281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15654.532863] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15661.380776] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15676.189449] loop0: detected capacity change from 0 to 204800000 [15676.350211] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15676.364538] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15676.386715] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15683.195654] loop0: detected capacity change from 0 to 204800000 [15683.446415] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15683.492665] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15683.511964] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15689.262106] loop0: detected capacity change from 0 to 204800000 [15695.912420] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15696.269255] Lustre: fs15246-OST0000: new disk, initializing [15696.276346] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15696.286311] Lustre: 514618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15696.421084] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15704.159945] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15715.170989] loop1: detected capacity change from 0 to 204800000 [15721.460848] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15721.637649] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15721.709638] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15721.800609] Lustre: fs15246-MDT0000: new disk, initializing [15722.031109] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15722.064675] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15722.206141] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15723.237032] Lustre: 515742:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005c459c53 x1874642429068160/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787798478 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15727.596653] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15731.374473] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15731.389618] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15731.423100] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15753.194474] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15753.204570] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15755.157478] Lustre: server umount fs15246-MDT0000 complete [15758.316057] LustreError: 512207:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15763.444186] LustreError: 516165:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15763.914890] Lustre: server umount fs15246-OST0000 complete [15775.713512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15775.722513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15775.735104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15776.738142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15776.746534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15776.757158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15778.789749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15781.402226] Lustre: server umount lustre-OST0000 complete [15786.976803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15786.981500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15786.995273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15787.004331] Lustre: Skipped 1 previous similar message [15799.264132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15799.268782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15799.284904] Lustre: Skipped 5 previous similar messages [15799.632221] Lustre: server umount lustre-MDT0000 complete [15803.361863] LustreError: 511022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15803.399982] LustreError: 511022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15804.254225] LustreError: 511002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798544 with bad export cookie 2000247927825246905 [15804.276202] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15804.278267] LustreError: 511002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15804.970869] Lustre: server umount lustre-MDT0001 complete [15818.383824] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [15821.193935] Key type lgssc unregistered [15821.602657] LNet: 518846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15821.608839] LNetError: 518846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15821.649474] LNet: Removed LNI 192.168.201.114@tcp [15822.734453] Key type .llcrypt unregistered [15822.737965] Key type ._llcrypt unregistered [15843.224519] Key type ._llcrypt registered [15843.225978] Key type .llcrypt registered [15843.304429] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [15844.192325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15844.236088] alg: No test for adler32 (adler32-zlib) [15845.487136] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [15846.025374] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [15847.879184] Key type lgssc registered [15849.467720] Lustre: Echo OBD driver; http://www.lustre.org/ [15889.974965] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [15901.779329] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [15940.880560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15940.924962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15942.249114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15942.271547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15942.365650] Lustre: lustre-MDT0000: new disk, initializing [15942.453155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15942.466279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15947.407582] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15957.950911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15958.064487] Lustre: 523315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15958.164507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15958.174635] Lustre: Skipped 1 previous similar message [15958.268564] Lustre: lustre-MDT0001: new disk, initializing [15958.362530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15958.391697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15958.398427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15962.464569] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15971.186050] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15976.538404] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15983.441172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15983.835279] Lustre: lustre-OST0000: new disk, initializing [15983.842669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15983.849792] Lustre: 525279:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15983.973814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15990.768775] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [15993.921353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15993.930801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15994.059801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15998.993100] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16004.109805] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16004.343960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16008.914771] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16009.141604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16014.829159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16014.836715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16014.840703] LustreError: Skipped 1 previous similar message [16014.843593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16014.884424] Lustre: Skipped 1 previous similar message [16017.122451] Lustre: server umount lustre-OST0000 complete [16025.058031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16025.059438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16025.071820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16025.071828] Lustre: Skipped 1 previous similar message [16030.176451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16030.184939] Lustre: Skipped 2 previous similar messages [16034.275915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16034.289112] Lustre: Skipped 1 previous similar message [16035.295147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16035.431947] Lustre: server umount lustre-MDT0000 complete [16038.764070] LustreError: 523306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798778 with bad export cookie 17441259442140177891 [16038.773249] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16038.780298] LustreError: 523306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16039.190720] Lustre: server umount lustre-MDT0001 complete [16063.683387] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [16072.593975] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [16101.688911] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15778 sec ========= 22:47:18 (1787798838) [16103.451332] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:47:20 (1787798840) === [16107.283943] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:47:24 (1787798844) === [16146.151345] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [16149.211197] Key type lgssc unregistered [16149.591099] LNet: 531929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16149.600557] LNetError: 531929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16150.630211] LNet: Removed LNI 192.168.201.114@tcp [16151.897816] Key type .llcrypt unregistered [16151.899722] Key type ._llcrypt unregistered