[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479881035 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002227] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007673] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.012079] LSM: Security Framework initializing [ 0.013069] Yama: becoming mindful. [ 0.014041] SELinux: Initializing. [ 0.015081] *** VALIDATE selinux *** [ 0.024687] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030330] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033131] *** VALIDATE tmpfs *** [ 0.034506] *** VALIDATE proc *** [ 0.035302] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038176] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.044094] debug: unmapping init [mem 0xffffffff9fa59000-0xffffffff9fa60fff] [ 0.047270] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048817] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049027] ... version: 2 [ 0.049717] ... bit width: 48 [ 0.050009] ... generic registers: 4 [ 0.050730] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.051895] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053214] rcu: Hierarchical SRCU implementation. [ 0.055107] smp: Bringing up secondary CPUs ... [ 0.056389] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.062010] smp: Brought up 1 node, 4 CPUs [ 0.063999] smpboot: Max logical packages: 1 [ 0.064994] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105178] node 0 deferred pages initialised in 39ms [ 0.108392] devtmpfs: initialized [ 0.109183] x86/mm: Memory block size: 128MB [ 0.111131] gcov: version magic: 0x41383552 [ 0.112538] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.113098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.114226] pinctrl core: initialized pinctrl subsystem [ 0.115141] [ 0.115523] ************************************************************* [ 0.116010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.117020] ** ** [ 0.118006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.119007] ** ** [ 0.120007] ** This means that this kernel is built to expose internal ** [ 0.121006] ** IOMMU data structures, which may compromise security on ** [ 0.122009] ** your system. ** [ 0.123007] ** ** [ 0.124009] ** If you see this message and you are not debugging the ** [ 0.125008] ** kernel, report this immediately to your vendor! ** [ 0.126007] ** ** [ 0.127008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.128007] ************************************************************* [ 0.129510] NET: Registered protocol family 16 [ 0.130323] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.131040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.132050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.134099] cpuidle: using governor menu [ 0.135435] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.136355] PCI: Using configuration type 1 for base access [ 0.137118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.142175] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.143039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.145119] cryptd: max_cpu_qlen set to 1000 [ 0.149295] ACPI: Added _OSI(Module Device) [ 0.150015] ACPI: Added _OSI(Processor Device) [ 0.152014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.154014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.158980] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.165071] ACPI: Interpreter enabled [ 0.165933] ACPI: PM: (supports S0 S3 S4 S5) [ 0.166015] ACPI: Using IOAPIC for interrupt routing [ 0.167132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.168339] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.176240] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.177032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.178013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.179089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.185622] acpiphp: Slot [2] registered [ 0.187140] acpiphp: Slot [5] registered [ 0.189143] acpiphp: Slot [6] registered [ 0.191139] acpiphp: Slot [7] registered [ 0.192532] acpiphp: Slot [8] registered [ 0.195143] acpiphp: Slot [9] registered [ 0.196136] acpiphp: Slot [10] registered [ 0.198113] acpiphp: Slot [3] registered [ 0.199126] acpiphp: Slot [4] registered [ 0.201181] acpiphp: Slot [11] registered [ 0.202093] acpiphp: Slot [12] registered [ 0.204085] acpiphp: Slot [13] registered [ 0.205179] acpiphp: Slot [14] registered [ 0.206098] acpiphp: Slot [15] registered [ 0.207061] acpiphp: Slot [16] registered [ 0.208123] acpiphp: Slot [17] registered [ 0.210097] acpiphp: Slot [18] registered [ 0.211070] acpiphp: Slot [19] registered [ 0.212000] acpiphp: Slot [20] registered [ 0.212000] acpiphp: Slot [21] registered [ 0.213064] acpiphp: Slot [22] registered [ 0.214059] acpiphp: Slot [23] registered [ 0.215105] acpiphp: Slot [24] registered [ 0.217116] acpiphp: Slot [25] registered [ 0.219106] acpiphp: Slot [26] registered [ 0.220107] acpiphp: Slot [27] registered [ 0.222104] acpiphp: Slot [28] registered [ 0.225163] acpiphp: Slot [29] registered [ 0.226099] acpiphp: Slot [30] registered [ 0.228097] acpiphp: Slot [31] registered [ 0.229053] PCI host bridge to bus 0000:00 [ 0.231020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.233023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.236027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.238022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.241024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.244032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.246085] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.247799] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.250000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.257017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.262728] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.264016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.265015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.267014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.269533] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.271585] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.274052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.276965] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.282017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.290907] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.296027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.301613] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.309018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.317019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.337021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.348000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.354020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.361020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.375019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.382229] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.387013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.392019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.405025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.415615] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.427018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.434016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.455017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.466717] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.485020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.498017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.513015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.523757] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.531016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.538018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.556017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.564564] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.567344] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.569314] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.571281] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.573188] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.577125] iommu: Default domain type: Passthrough [ 0.580329] SCSI subsystem initialized [ 0.581089] ACPI: bus type USB registered [ 0.583088] usbcore: registered new interface driver usbfs [ 0.584094] usbcore: registered new interface driver hub [ 0.586087] usbcore: registered new device driver usb [ 0.588179] pps_core: LinuxPPS API ver. 1 registered [ 0.589010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.592054] PTP clock support registered [ 0.594128] EDAC MC: Ver: 3.0.0 [ 0.596134] PCI: Using ACPI for IRQ routing [ 0.597821] NetLabel: Initializing [ 0.599011] NetLabel: domain hash size = 128 [ 0.600009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.602088] NetLabel: unlabeled traffic allowed by default [ 0.604118] vgaarb: loaded [ 0.605264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.607032] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.616129] clocksource: Switched to clocksource kvm-clock [ 0.701352] VFS: Disk quotas dquot_6.6.0 [ 0.702366] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.703844] *** VALIDATE ramfs *** [ 0.704587] *** VALIDATE hugetlbfs *** [ 0.705544] pnp: PnP ACPI init [ 0.707271] pnp: PnP ACPI: found 6 devices [ 0.720802] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.723378] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.724651] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.725875] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.727291] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.729017] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.730768] NET: Registered protocol family 2 [ 0.732693] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.736356] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.738886] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.743689] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.746694] TCP: Hash tables configured (established 65536 bind 65536) [ 0.749606] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.752829] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.755656] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.758565] NET: Registered protocol family 1 [ 0.761136] RPC: Registered named UNIX socket transport module. [ 0.763243] RPC: Registered udp transport module. [ 0.764510] RPC: Registered tcp transport module. [ 0.765661] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.767780] NET: Registered protocol family 44 [ 0.769434] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.771440] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.773314] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.775602] PCI: CLS 0 bytes, default 64 [ 0.777175] Unpacking initramfs... [ 2.168568] debug: unmapping init [mem 0xffff8e1a7cc54000-0xffff8e1a7ffbffff] [ 2.172638] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.174990] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.177851] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.658031] Initialise system trusted keyrings [ 2.659908] Key type blacklist registered [ 2.662654] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.675137] zbud: loaded [ 2.678567] *** VALIDATE nfs *** [ 2.680062] *** VALIDATE nfs4 *** [ 2.681914] pstore: using deflate compression [ 2.686124] Platform Keyring initialized [ 2.813902] NET: Registered protocol family 38 [ 2.815714] Key type asymmetric registered [ 2.817610] Asymmetric key parser 'x509' registered [ 2.819537] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.822847] io scheduler mq-deadline registered [ 2.824901] io scheduler kyber registered [ 2.826702] io scheduler bfq registered [ 2.828765] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.832176] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.835408] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.837501] ACPI: Power Button [PWRF] [ 2.842658] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.849024] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.868474] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.875835] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.890304] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.918551] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.945558] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.951145] Non-volatile memory driver v1.3 [ 2.953240] Linux agpgart interface v0.103 [ 2.981536] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.983884] vda: detected capacity change from 0 to 74706944 [ 2.998818] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.001390] vdb: detected capacity change from 0 to 1073741824 [ 3.016401] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.019079] vdc: detected capacity change from 0 to 2621440000 [ 3.032609] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.035146] vdd: detected capacity change from 0 to 2621440000 [ 3.050230] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.053104] vde: detected capacity change from 0 to 4294967296 [ 3.068301] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.071778] vdf: detected capacity change from 0 to 4294967296 [ 3.080966] libphy: Fixed MDIO Bus: probed [ 3.099195] usbcore: registered new interface driver usbserial_generic [ 3.101768] usbserial: USB Serial support registered for generic [ 3.104483] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.109375] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.111351] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.113286] mousedev: PS/2 mouse device common for all mice [ 3.114995] rtc_cmos 00:05: RTC can wake from S4 [ 3.116652] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.117451] rtc_cmos 00:05: registered as rtc0 [ 3.122285] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.122480] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.126767] intel_pstate: CPU model not supported [ 3.127151] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.131694] hid: raw HID events driver (C) Jiri Kosina [ 3.133089] usbcore: registered new interface driver usbhid [ 3.135123] usbhid: USB HID core driver [ 3.136737] drop_monitor: Initializing network drop monitor service [ 3.138656] Initializing XFRM netlink socket [ 3.140122] NET: Registered protocol family 10 [ 3.143493] Segment Routing with IPv6 [ 3.145194] NET: Registered protocol family 17 [ 3.147412] mpls_gso: MPLS GSO support [ 3.151474] RAS: Correctable Errors collector initialized. [ 3.152987] AVX version of gcm_enc/dec engaged. [ 3.153873] AES CTR mode by8 optimization enabled [ 3.218402] sched_clock: Marking stable (3218378952, 0)->(4153594588, -935215636) [ 3.223910] registered taskstats version 1 [ 3.225747] Loading compiled-in X.509 certificates [ 3.227657] zswap: loaded using pool lzo/zbud [ 3.251621] Key type big_key registered [ 3.263694] Key type encrypted registered [ 3.265322] ima: No TPM chip found, activating TPM-bypass! [ 3.267520] ima: Allocated hash algorithm: sha1 [ 3.269341] ima: No architecture policies found [ 3.271265] evm: Initialising EVM extended attributes: [ 3.274346] evm: security.selinux [ 3.275489] evm: security.ima [ 3.276555] evm: security.capability [ 3.277720] evm: HMAC attrs: 0x1 [ 3.279300] rtc_cmos 00:05: setting system clock to 2026-08-11 07:01:05 UTC (1786431665) [ 3.283491] debug: unmapping init [mem 0xffffffffa0a03000-0xffffffffa0bfffff] [ 3.285374] debug: unmapping init [mem 0xffffffff9f782000-0xffffffff9fa58fff] [ 3.293738] Write protecting the kernel read-only data: 28672k [ 3.296985] debug: unmapping init [mem 0xffffffff9de03000-0xffffffff9dffffff] [ 3.299228] debug: unmapping init [mem 0xffffffff9e714000-0xffffffff9e7fffff] [ 3.327152] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.335310] systemd[1]: Detected virtualization kvm. [ 3.337410] systemd[1]: Detected architecture x86-64. [ 3.338656] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.363552] systemd[1]: No hostname configured. [ 3.365249] systemd[1]: Set hostname to . [ 3.367095] random: systemd: uninitialized urandom read (16 bytes read) [ 3.369642] systemd[1]: Initializing machine ID from random generator. [ 3.398215] random: ln: uninitialized urandom read (6 bytes read) [ 3.494757] random: systemd: uninitialized urandom read (16 bytes read) [ 3.497445] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.501595] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.505553] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.159928] device-mapper: uevent: version 1.0.3 [ 4.162450] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.874730] random: fast init done [ 4.932374] virtio_net virtio0 ens2: renamed from eth0 [ 4.993223] scsi host0: ata_piix [ 5.030044] scsi host1: ata_piix [ 5.038105] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.040574] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.665404] dracut-initqueue[569]: RTNETLINK answers: File exists [ 9.782358] random: crng init done [ 9.784010] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.211132] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.451160] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.718486] SELinux: Disabled at runtime. [ 11.782805] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.791062] systemd[1]: Detected virtualization kvm. [ 11.793034] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.245052] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.248587] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.254153] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.258066] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.261538] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.268218] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.273314] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.362264] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.708132] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.029210] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.090066] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.181018] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.233431] EDAC sbridge: Ver: 1.1.2 [ 14.932834] Key type dns_resolver registered [ 15.239734] NFS: Registering the id_resolver key type [ 15.241754] Key type id_resolver registered [ 15.243462] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg111-server login: [ 55.084359] libcfs: loading out-of-tree module taints kernel. [ 55.159994] Key type ._llcrypt registered [ 55.165385] Key type .llcrypt registered [ 55.310857] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 81.067353] hrtimer: interrupt took 3344278 ns [ 81.269135] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 83.134919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.145623] alg: No test for adler32 (adler32-zlib) [ 84.593597] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 85.650372] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 87.528574] Key type lgssc registered [ 90.032097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.471379] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 153.045356] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 166.535363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.565653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.961524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 168.019708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 168.200980] Lustre: lustre-MDT0000: new disk, initializing [ 168.330279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.364856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 173.643859] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.699984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.835349] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 186.867279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 186.869275] Lustre: Skipped 1 previous similar message [ 186.946407] Lustre: lustre-MDT0001: new disk, initializing [ 187.011768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.032395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.040087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.668944] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.375152] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.382654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.609627] Lustre: lustre-OST0000: new disk, initializing [ 209.613359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.623389] Lustre: 8442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.690100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.734709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.737369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.789396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 214.584105] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.797460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.948984] Lustre: lustre-OST0001: new disk, initializing [ 228.952858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 228.959634] Lustre: 9515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.064239] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.625639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.670792] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.819423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.877719] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.583069] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 262.462702] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 269.347495] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.221961] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing yml_node [ 279.024636] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 281.346961] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 283.932893] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 285.773313] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 11 03:05:46 EDT 2026 [ 305.844820] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 307.701540] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 325.600754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 325.607347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.615825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.161553] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.162288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.181516] Lustre: Skipped 2 previous similar messages [ 328.186165] Lustre: Skipped 2 previous similar messages [ 330.855966] Lustre: server umount lustre-MDT0000 complete [ 335.214224] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786431997 with bad export cookie 3112305992999100481 [ 335.222040] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.225973] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 338.401293] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.402628] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.424445] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 338.450288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.452912] Lustre: Skipped 1 previous similar message [ 340.449110] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.458699] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 341.628968] Lustre: server umount lustre-MDT0001 complete [ 345.498519] Lustre: server umount lustre-OST0000 complete [ 349.502234] Lustre: server umount lustre-OST0001 complete [ 356.530354] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 367.977360] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 401.888665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.088763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 402.110748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 402.188445] Lustre: lustre-MDT0000: new disk, initializing [ 402.252969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.266969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 405.587478] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 417.114364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.308119] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.390846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.400477] Lustre: Skipped 1 previous similar message [ 417.640961] Lustre: lustre-MDT0001: new disk, initializing [ 417.843784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.894879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 417.915559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 424.078071] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 435.390047] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 442.140791] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 449.173659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 449.342531] Lustre: lustre-OST0000: new disk, initializing [ 449.345699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 449.350541] Lustre: 17037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 449.425917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.282384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 451.292703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 451.413561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.768135] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 465.839706] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 470.492180] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 470.799135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.422542] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.743596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 482.310180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 482.320043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.325690] LustreError: Skipped 1 previous similar message [ 482.336907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 482.355931] Lustre: Skipped 2 previous similar messages [ 482.395712] Lustre: Skipped 3 previous similar messages [ 484.695158] Lustre: server umount lustre-OST0000 complete [ 492.513708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.518171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.535710] Lustre: Skipped 1 previous similar message [ 492.546635] Lustre: Skipped 1 previous similar message [ 496.096496] Lustre: server umount lustre-MDT0000 complete [ 497.633333] LustreError: 15079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.647679] LustreError: 15079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 499.146749] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786432161 with bad export cookie 3112305992999102014 [ 499.159162] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.186276] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 499.579462] Lustre: server umount lustre-MDT0001 complete [ 503.951629] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:09:24 (1786432164) [ 506.191215] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 508.446853] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:09:28 (1786432168) [ 517.309569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.936170] LustreError: 19445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.087199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.188554] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 523.241761] LustreError: 19446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.276298] LustreError: 19446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 531.918271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.801916] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 544.611976] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 550.379775] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 557.536967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 565.409118] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 572.760434] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 587.753660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.766732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 587.778371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 587.810945] Lustre: Skipped 1 previous similar message [ 590.253882] Lustre: server umount lustre-OST0000 complete [ 601.130226] Lustre: server umount lustre-MDT0000 complete [ 603.117804] LustreError: 19442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.144977] LustreError: 19442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 605.332789] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786432267 with bad export cookie 3112305992999102896 [ 605.355241] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 605.637698] Lustre: server umount lustre-MDT0001 complete [ 620.869824] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 623.927534] Key type lgssc unregistered [ 624.312798] LNet: 22981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.326286] LNetError: 22981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.337265] LNet: Removed LNI 192.168.201.111@tcp [ 625.380223] Key type .llcrypt unregistered [ 625.383234] Key type ._llcrypt unregistered [ 651.854795] Key type ._llcrypt registered [ 651.857494] Key type .llcrypt registered [ 651.966401] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 653.485391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 653.587140] alg: No test for adler32 (adler32-zlib) [ 654.852370] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 655.222846] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 656.904202] Key type lgssc registered [ 658.659494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 668.218694] loop: module loaded [ 668.299248] loop0: detected capacity change from 0 to 204800000 [ 668.565986] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 668.585410] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 668.955643] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 675.800553] loop0: detected capacity change from 0 to 204800000 [ 676.113484] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.134177] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.155661] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.726349] loop0: detected capacity change from 0 to 204800000 [ 689.651266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 689.686957] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.159200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 691.226183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 691.378304] Lustre: lustre-MDT0000: new disk, initializing [ 691.570790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.616243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 696.991221] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 706.025133] loop1: detected capacity change from 0 to 204800000 [ 712.317814] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 712.650643] Lustre: lustre-OST07c6: new disk, initializing [ 712.656098] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 712.669568] Lustre: Skipped 1 previous similar message [ 712.676308] Lustre: 26170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 712.774464] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 718.441795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 718.460477] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 718.502644] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 719.717829] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 724.290092] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 724.296477] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 724.300149] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 724.303444] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 724.307025] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 724.310622] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 742.980318] Lustre: 26785:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 742.987657] Lustre: 26785:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 743.006311] Lustre: 26785:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 743.022601] Lustre: 26785:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 743.041230] Lustre: 26785:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 743.051450] Lustre: 26785:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.921071] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 747.940270] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 747.947180] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 747.953978] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.968063] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.976594] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.988744] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 747.996187] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.003796] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 748.018892] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.028387] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 748.036682] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.739504] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 752.755056] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 752.769328] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 752.780647] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.786224] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.791701] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.801576] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 752.805934] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.810307] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 752.813724] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.817647] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.821247] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.546940] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 760.578344] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 760.595838] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.616463] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 760.632368] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.645271] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 760.667243] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 760.681937] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 760.694865] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 760.706256] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 760.719204] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.729137] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.190937] Lustre: 26785:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 772.212419] Lustre: 26785:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 772.226378] Lustre: 26785:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.234652] Lustre: 26785:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.247760] Lustre: 26785:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.261380] Lustre: 26785:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.269133] Lustre: 26785:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 772.274256] Lustre: 26785:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.283806] Lustre: 26785:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.290789] Lustre: 26785:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.303516] Lustre: 26785:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.307451] Lustre: 26785:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 815.588343] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 815.607150] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.670318] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 820.584896] Lustre: server umount lustre-OST07c6 complete [ 830.858645] Lustre: server umount lustre-MDT0000 complete [ 849.369386] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 853.724427] Key type lgssc unregistered [ 854.175926] LNet: 28735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.188077] LNetError: 28735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.215389] LNet: Removed LNI 192.168.201.111@tcp [ 855.488135] Key type .llcrypt unregistered [ 855.490238] Key type ._llcrypt unregistered [ 878.079986] Key type ._llcrypt registered [ 878.086187] Key type .llcrypt registered [ 878.302123] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 879.178942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 879.217920] alg: No test for adler32 (adler32-zlib) [ 880.498599] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 880.761548] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 882.440253] Key type lgssc registered [ 883.762715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.405775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.420699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.974025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.224959] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 904.164365] LustreError: 30005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.207463] LustreError: 30005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 908.087703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.450596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 913.363337] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 921.570304] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 928.223431] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 935.643752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 936.091024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 939.131231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 943.051809] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 951.427487] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 969.187483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.200295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.218360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 972.977505] Lustre: server umount lustre-OST0000 complete [ 979.428661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.434924] Lustre: Skipped 1 previous similar message [ 979.438092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.442345] Lustre: Skipped 2 previous similar messages [ 983.377106] Lustre: server umount lustre-MDT0000 complete [ 984.555079] LustreError: 32001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.569382] LustreError: 32001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 988.034163] LustreError: 32701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786432650 with bad export cookie 5736738135285624604 [ 988.056601] LustreError: 32701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 988.061110] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 988.646508] Lustre: server umount lustre-MDT0001 complete [ 1000.642982] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1003.812989] Key type lgssc unregistered [ 1004.136227] LNet: 33747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.159283] LNetError: 33747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.226148] LNet: Removed LNI 192.168.201.111@tcp [ 1006.455964] Key type .llcrypt unregistered [ 1006.463869] Key type ._llcrypt unregistered [ 1024.312186] Key type ._llcrypt registered [ 1024.316178] Key type .llcrypt registered [ 1024.386549] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1025.570842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1025.597981] alg: No test for adler32 (adler32-zlib) [ 1026.682948] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1026.853743] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1028.472214] Key type lgssc registered [ 1029.787595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1037.495026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1037.520122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.165735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.359925] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1049.584193] LustreError: 35018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1049.623654] LustreError: 35018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1052.962932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1053.427475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1057.657809] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1064.719663] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1070.544920] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1078.413115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.744214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.902311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1085.371399] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1094.603773] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1108.464183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.483438] Lustre: Skipped 1 previous similar message [ 1108.486902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.490725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1113.573985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1113.592719] Lustre: Skipped 2 previous similar messages [ 1114.811374] Lustre: server umount lustre-OST0000 complete [ 1120.227286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1120.258277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.282456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.810142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.817574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.834186] Lustre: Skipped 1 previous similar message [ 1127.291706] Lustre: server umount lustre-MDT0000 complete [ 1128.934849] LustreError: 37620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.961584] LustreError: 37620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1130.942232] LustreError: 34998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786432793 with bad export cookie 2093658132304936056 [ 1130.947252] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.949263] LustreError: 34998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1131.161811] Lustre: server umount lustre-MDT0001 complete [ 1142.537279] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1145.891526] Key type lgssc unregistered [ 1146.372818] LNet: 38581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.377413] LNetError: 38581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.433148] LNet: Removed LNI 192.168.201.111@tcp [ 1149.010078] Key type .llcrypt unregistered [ 1149.019435] Key type ._llcrypt unregistered [ 1176.154941] Key type ._llcrypt registered [ 1176.160643] Key type .llcrypt registered [ 1176.404756] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1177.995962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1178.060815] alg: No test for adler32 (adler32-zlib) [ 1179.221277] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1179.599115] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1181.416392] Key type lgssc registered [ 1182.592276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1189.361424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1189.378582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.904720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.463399] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1201.123819] LustreError: 39998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.142752] LustreError: 39998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1204.036588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.502122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1209.873245] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1216.721674] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1222.715504] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1229.549232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1229.982239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.210865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1236.516876] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1244.789457] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1255.932208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1255.942175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.966305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.073282] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 1260.002122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.018657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1260.041486] Lustre: Skipped 1 previous similar message [ 1260.045305] Lustre: Skipped 1 previous similar message [ 1261.392997] Lustre: server umount lustre-MDT0000 complete [ 1262.079170] LustreError: 42122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.123730] LustreError: 39994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.140854] LustreError: 39994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1265.765017] LustreError: 40734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786432927 with bad export cookie 6674410182842521433 [ 1265.790650] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.274077] LustreError: 39995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.314744] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.11@tcp (stopping) [ 1267.321307] Lustre: Skipped 1 previous similar message [ 1270.251769] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.433980] Lustre: server umount lustre-MDT0001 complete [ 1293.565971] Lustre: server umount lustre-OST0000 complete [ 1310.655807] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1313.289365] Key type lgssc unregistered [ 1313.528660] LNet: 43863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.541693] LNetError: 43863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.560474] LNet: Removed LNI 192.168.201.111@tcp [ 1314.256768] Key type .llcrypt unregistered [ 1314.258246] Key type ._llcrypt unregistered [ 1337.047813] Key type ._llcrypt registered [ 1337.050697] Key type .llcrypt registered [ 1337.233594] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1338.825034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1339.057196] alg: No test for adler32 (adler32-zlib) [ 1340.318120] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1340.568578] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1342.336284] Key type lgssc registered [ 1343.673162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.711823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1351.754431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.343159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.129677] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1362.932579] LustreError: 45280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.954403] LustreError: 45280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1367.353534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.742509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.687133] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1381.245412] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1388.104603] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1395.694182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.052901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.128883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1401.632833] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1408.890750] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1418.723219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1418.736694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.771477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.921363] Lustre: lustre-OST0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 1422.821413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1422.827617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.828579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1425.983974] Lustre: lustre-OST0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 1425.990488] Lustre: Skipped 1 previous similar message [ 1431.109829] Lustre: lustre-OST0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 1431.112867] Lustre: Skipped 2 previous similar messages [ 1432.032391] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1432.214496] Lustre: server umount lustre-OST0000 complete [ 1450.152501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.162734] Lustre: Skipped 1 previous similar message [ 1450.163206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.177843] Lustre: Skipped 1 previous similar message [ 1456.701482] Lustre: server umount lustre-MDT0000 complete [ 1458.662413] LustreError: 46763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.691676] LustreError: 46763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1460.083855] LustreError: 45261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786433122 with bad export cookie 3528486736737973040 [ 1460.098024] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1460.099668] LustreError: 45261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1466.553595] Lustre: server umount lustre-MDT0001 complete [ 1476.212506] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1478.713233] Key type lgssc unregistered [ 1479.006529] LNet: 48999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.010836] LNetError: 48999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.026120] LNet: Removed LNI 192.168.201.111@tcp [ 1479.843190] Key type .llcrypt unregistered [ 1479.845095] Key type ._llcrypt unregistered [ 1502.924978] Key type ._llcrypt registered [ 1502.928054] Key type .llcrypt registered [ 1503.047423] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1504.460982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1504.490072] alg: No test for adler32 (adler32-zlib) [ 1505.550385] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1505.829988] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1507.592168] Key type lgssc registered [ 1508.504721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1515.343753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1515.386988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.923915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.720237] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1527.289373] LustreError: 50415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.318726] LustreError: 50415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1529.987325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.375433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1535.345696] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1543.507657] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1550.074738] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1558.252857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1558.533380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.694133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1565.310502] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1573.109034] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1578.905358] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1582.270636] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1594.466696] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-11 3h27m34s [ 1596.170600] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1600.995311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.998455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.011712] Lustre: Skipped 1 previous similar message [ 1601.025617] Lustre: Skipped 2 previous similar messages [ 1602.085564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1602.097360] Lustre: Skipped 3 previous similar messages [ 1607.234253] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 1607.242628] Lustre: Skipped 1 previous similar message [ 1612.256138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1612.511855] Lustre: server umount lustre-MDT0000 complete [ 1613.421205] LustreError: 50410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1615.932363] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1619.520611] LustreError: 50409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1619.531775] LustreError: 50409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1622.707100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.873462] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.095958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.145867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1624.125854] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1627.982924] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1631.683950] Lustre: DEBUG MARKER: Wait for df (36691) ... [ 1633.322186] Lustre: DEBUG MARKER: done [ 1636.833481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1636.838829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.845992] Lustre: Skipped 1 previous similar message [ 1636.853097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1636.857147] Lustre: Skipped 3 previous similar messages [ 1638.369518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1638.384511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.843092] Lustre: server umount lustre-OST0000 complete [ 1649.635923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.650927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1649.660242] Lustre: Skipped 4 previous similar messages [ 1661.408908] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1661.620266] Lustre: server umount lustre-MDT0000 complete [ 1662.954872] LustreError: 50414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.970928] LustreError: 50414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1665.529221] LustreError: 52258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786433327 with bad export cookie 1066418998753875894 [ 1665.554399] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1666.002273] Lustre: server umount lustre-MDT0001 complete [ 1676.425497] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1679.076152] Key type lgssc unregistered [ 1679.480882] LNet: 55341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.488170] LNetError: 55341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.503450] LNet: Removed LNI 192.168.201.111@tcp [ 1680.248703] Key type .llcrypt unregistered [ 1680.251236] Key type ._llcrypt unregistered [ 1717.937211] Key type ._llcrypt registered [ 1717.942121] Key type .llcrypt registered [ 1718.091174] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1719.270459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1719.285215] alg: No test for adler32 (adler32-zlib) [ 1720.387551] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1720.669474] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1722.458119] Key type lgssc registered [ 1723.913330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1732.164864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1732.184813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.417159] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1733.432574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1733.549359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.072937] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1746.549759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.641340] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1746.678431] Lustre: 57556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1747.048822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1750.974877] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1757.637540] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1763.221168] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1769.610361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1769.806337] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1770.046718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.387316] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1778.862971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1782.935251] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1789.264260] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-11 3h30m50s [ 1790.733633] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1797.449835] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1800.045233] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1800.111721] LustreError: 57562:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1820.274543] Lustre: lustre-MDT0000: Client 852c9bf4-3010-4148-acef-38ee28d9dcd0 (at 192.168.201.11@tcp) reconnecting [ 1840.192100] LustreError: 57562:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1841.788444] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1845.074623] Lustre: DEBUG MARKER: done [ 1853.323584] Lustre: server umount lustre-OST0000 complete [ 1859.553045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1859.561657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.572390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.642953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.649567] Lustre: Skipped 1 previous similar message [ 1866.726425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.737036] Lustre: Skipped 1 previous similar message [ 1867.916384] Lustre: server umount lustre-MDT0000 complete [ 1870.817383] Lustre: 56877:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786433517/real 1786433517] req@0000000090f50ea6 x1873211170019200/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786433533 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1870.847676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.861394] Lustre: Skipped 1 previous similar message [ 1871.906139] LustreError: 59013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.927934] LustreError: 59013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1871.980597] LustreError: 57548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786433534 with bad export cookie 4651229353271659394 [ 1871.982997] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.999225] LustreError: 57548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1872.448566] Lustre: server umount lustre-MDT0001 complete [ 1884.477547] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1887.500903] Key type lgssc unregistered [ 1887.826995] LNet: 61888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.846416] LNetError: 61888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.865810] LNet: Removed LNI 192.168.201.111@tcp [ 1888.783256] Key type .llcrypt unregistered [ 1888.785081] Key type ._llcrypt unregistered [ 1934.423122] Key type ._llcrypt registered [ 1934.425179] Key type .llcrypt registered [ 1934.575905] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1936.272243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.298962] alg: No test for adler32 (adler32-zlib) [ 1937.461703] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 1937.768591] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1939.504180] Key type lgssc registered [ 1940.988805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.402886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.454757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.887351] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1945.904943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1946.094450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.980834] Lustre: Failing over lustre-MDT0000 [ 1947.167541] Lustre: server umount lustre-MDT0000 complete [ 1953.210915] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:33:33 (1786433613) [ 1959.259164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.577373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.578242] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1972.862691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.970189] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1973.050488] Lustre: 64687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1973.322079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1977.797588] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1985.117089] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1991.121826] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1998.430869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1998.558460] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1998.697773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.747886] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2005.641546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2011.964897] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2017.857823] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2024.417250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2024.424205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.458556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2024.602934] Lustre: server umount lustre-MDT0000 complete [ 2026.486880] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2028.370622] LustreError: 64680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786433690 with bad export cookie 7499720144374765396 [ 2028.379380] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.383214] LustreError: 64680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2028.742754] Lustre: server umount lustre-MDT0001 complete [ 2032.177584] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2036.395299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2045.002852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.944511] Lustre: 63611:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786433693/real 1786433693] req@000000000cf37d4c x1873211397544192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786433709 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2046.969519] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.984391] Lustre: Skipped 1 previous similar message [ 2048.993428] Lustre: 63612:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786433695/real 1786433695] req@00000000e2b718ee x1873211397544320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786433711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2052.064210] Lustre: 63613:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786433698/real 1786433698] req@00000000c8f138a2 x1873211397544448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786433714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2053.529732] LustreError: 66535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.551622] LustreError: 66535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2053.642552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.748270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2054.626350] Lustre: 63611:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786433700/real 1786433700] req@00000000016f6b9c x1873211397544704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786433716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2057.706182] LustreError: 68456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.719559] LustreError: 68456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2058.236190] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2059.812451] Lustre: 63611:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786433706/real 1786433706] req@00000000b3170b7e x1873211397544832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786433722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2062.820673] LustreError: 68456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.844297] LustreError: 68456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2065.896842] LustreError: 68453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.909311] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2065.934431] LustreError: 68453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2067.506706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.945403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2072.920734] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2073.089694] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2080.095328] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2086.861022] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2098.658129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2098.665835] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.679023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.700866] Lustre: Skipped 2 previous similar messages [ 2101.673085] Lustre: server umount lustre-MDT0000 complete [ 2103.778384] LustreError: 68456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.790803] LustreError: 68456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2105.278118] LustreError: 64681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786433767 with bad export cookie 7499720144374767678 [ 2105.287574] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2105.295488] LustreError: 64681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2105.510595] Lustre: server umount lustre-MDT0001 complete [ 2111.199596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2121.918626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.283392] Lustre: 63613:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786433771/real 1786433771] req@00000000dc9ff2f5 x1873211397572352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786433787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.322974] Lustre: 63613:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2125.352780] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.090440] LustreError: 69191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.264482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.401262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2137.548679] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2141.409198] Lustre: 63612:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786433787/real 1786433787] req@000000003eb490ad x1873211397573120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786433803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2141.447654] Lustre: 63612:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2146.851166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.436394] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2152.428690] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2154.502922] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2158.737154] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2164.451157] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2172.550831] Lustre: server umount lustre-MDT0000 complete [ 2172.898599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2172.907877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.917215] LustreError: 66535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.928940] LustreError: 66535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2176.351163] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786433838 with bad export cookie 7499720144374769057 [ 2176.354225] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2176.361209] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2176.745923] Lustre: server umount lustre-MDT0001 complete [ 2183.443748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2186.794349] Lustre: DEBUG MARKER: files compared the same [ 2199.401538] Lustre: server umount lustre-OST0000 complete [ 2218.480823] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2221.563670] Key type lgssc unregistered [ 2222.027875] LNet: 74419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.040950] LNetError: 74419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.064666] LNet: Removed LNI 192.168.201.111@tcp [ 2223.080958] Key type .llcrypt unregistered [ 2223.083234] Key type ._llcrypt unregistered [ 2247.666950] Key type ._llcrypt registered [ 2247.671233] Key type .llcrypt registered [ 2247.793347] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2248.937630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2249.040828] alg: No test for adler32 (adler32-zlib) [ 2250.183618] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 2250.520510] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2252.248171] Key type lgssc registered [ 2253.863669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2262.872443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2262.889661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.651670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.658889] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug +malloc all [ 2274.790239] LustreError: 75844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.811656] LustreError: 75844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2277.592926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.946107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.037502] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug +malloc all [ 2290.441307] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2297.330451] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2305.079382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2305.360638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.553428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2312.869255] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug +malloc all [ 2321.639785] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2331.111327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2331.136637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.181283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.153435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2333.161316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.165161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.186604] Lustre: Skipped 1 previous similar message [ 2336.738739] Lustre: server umount lustre-OST0000 complete [ 2343.393949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.396350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.406898] Lustre: Skipped 1 previous similar message [ 2343.421159] Lustre: Skipped 1 previous similar message [ 2348.516038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.518852] Lustre: Skipped 1 previous similar message [ 2353.633440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.638168] Lustre: Skipped 1 previous similar message [ 2355.168278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2355.388377] Lustre: server umount lustre-MDT0000 complete [ 2358.756099] LustreError: 75844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.786219] LustreError: 75844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2359.545440] LustreError: 75825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786434021 with bad export cookie 4685992700462106035 [ 2359.548010] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.552377] LustreError: 75825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2359.891934] Lustre: server umount lustre-MDT0001 complete [ 2372.055458] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2374.954487] Key type lgssc unregistered [ 2375.225727] LNet: 79384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2375.237513] LNetError: 79384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2376.294789] LNet: Removed LNI 192.168.201.111@tcp [ 2377.358643] Key type .llcrypt unregistered [ 2377.360113] Key type ._llcrypt unregistered [ 2401.709089] Key type ._llcrypt registered [ 2401.712599] Key type .llcrypt registered [ 2401.896660] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2403.224563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2403.256668] alg: No test for adler32 (adler32-zlib) [ 2404.467958] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 2404.774565] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2406.488210] Key type lgssc registered [ 2407.797097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2414.499732] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2414.515238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.112665] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2435.234523] LustreError: 80793:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.111@tcp: failed processing log, type 4: rc = -110 [ 2461.920428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.301439] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2476.757824] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2483.971833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.070596] Lustre: *** cfs_fail_loc=706, val=0*** [ 2484.458108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.529906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2489.399401] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2489.826468] LustreError: 80819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.847768] LustreError: 80819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2494.949324] LustreError: 80820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.997204] LustreError: 81879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.026523] LustreError: 81879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2498.099672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.409146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2503.434176] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2510.099752] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2515.057368] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2519.063168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.072767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2519.101719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.913090] Lustre: server umount lustre-OST0000 complete [ 2529.251098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2529.255248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.312347] Lustre: Skipped 1 previous similar message [ 2529.336914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.355616] Lustre: Skipped 2 previous similar messages [ 2534.372360] LustreError: 81878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.398819] LustreError: 81878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2534.783943] Lustre: server umount lustre-MDT0000 complete [ 2539.492371] LustreError: 81883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.522383] LustreError: 81883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2540.471073] LustreError: 80800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786434202 with bad export cookie 5643043451316168122 [ 2540.488864] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2540.495699] LustreError: 80800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2541.302599] Lustre: server umount lustre-MDT0001 complete [ 2552.572235] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2555.152956] Key type lgssc unregistered [ 2555.415847] LNet: 84411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2555.454974] LNetError: 84411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2555.493847] LNet: Removed LNI 192.168.201.111@tcp [ 2556.447395] Key type .llcrypt unregistered [ 2556.455615] Key type ._llcrypt unregistered [ 2582.123112] Key type ._llcrypt registered [ 2582.125251] Key type .llcrypt registered [ 2582.246298] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2583.302540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2583.446557] alg: No test for adler32 (adler32-zlib) [ 2584.499349] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 2584.803752] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2586.602090] Key type lgssc registered [ 2587.891132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2597.546618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2597.572204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.650310] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2611.642689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.485428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2617.830719] LustreError: 86519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.845462] LustreError: 86519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2617.973388] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2621.931158] LustreError: 86522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.966326] LustreError: 86522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2626.402223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2626.912677] LustreError: 86519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.951456] LustreError: 86519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2626.967684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.163107] LustreError: 86523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.183919] LustreError: 86523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2633.898609] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2637.287596] LustreError: 86517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.325474] LustreError: 86517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2642.364550] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2642.404722] Lustre: MGS: Client 0316331b-be60-4de6-abac-d3dc7e8f8c91 (at 0@lo) reconnecting [ 2642.429196] LustreError: 86533:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2642.687073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.789442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2648.284726] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2663.396631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.402118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2663.402906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2663.412924] Lustre: Skipped 1 previous similar message [ 2663.444514] Lustre: Skipped 1 previous similar message [ 2668.488728] Lustre: server umount lustre-OST0000 complete [ 2673.635225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2673.637189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.647562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.650036] Lustre: Skipped 1 previous similar message [ 2673.661626] Lustre: Skipped 1 previous similar message [ 2678.113977] Lustre: server umount lustre-MDT0000 complete [ 2678.756568] LustreError: 86519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.774352] LustreError: 86519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2681.812290] Lustre: server umount lustre-MDT0001 complete [ 2685.217502] Lustre: server umount MGS complete [ 2694.806447] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:45:55 (1786434355) [ 2723.825811] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 2734.686216] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2770.591225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.092340] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2786.677265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.044992] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2787.104743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2787.158233] Lustre: lustre-MDT0001: new disk, initializing [ 2787.230696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2787.254435] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2787.265604] Lustre: 93700:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2787.275948] LustreError: 93700:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2793.413291] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2806.252229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2806.558823] Lustre: lustre-OST0000: new disk, initializing [ 2806.563308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2806.569817] Lustre: 94467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2806.655294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.371655] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2816.097487] Lustre: 94895:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000098dab06e x1873212077022080/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786434494 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2823.855529] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2832.004493] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2832.021792] Lustre: MGS: Client 47fb4b0e-1690-41c7-95d2-8a21159af9b3 (at 0@lo) reconnecting [ 2832.075314] Lustre: 95513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2832.098041] LustreError: 93686:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2832.119063] LustreError: 93686:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2832.141823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2832.248818] Lustre: lustre-MDT0000: new disk, initializing [ 2832.415282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.447878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2832.468709] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2833.509713] Lustre: 95537:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000067ceb59c x1873212077028992/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786434511 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2838.051965] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2838.823510] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2842.676960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2842.688421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2842.893238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2872.296084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.313329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2872.318632] Lustre: Skipped 1 previous similar message [ 2876.267451] Lustre: server umount lustre-OST0000 complete [ 2882.529383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2882.532698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.544587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2882.593848] Lustre: Skipped 2 previous similar messages [ 2887.652889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.662253] Lustre: Skipped 2 previous similar messages [ 2894.304129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2894.436289] Lustre: server umount lustre-MDT0000 complete [ 2904.281469] Lustre: server umount lustre-MDT0001 complete [ 2907.773773] Lustre: server umount MGS complete [ 2919.446759] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:49:40 (1786434580) [ 2938.266767] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2941.022848] Key type lgssc unregistered [ 2941.268921] LNet: 98041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2941.272935] LNetError: 98041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.326689] LNet: Removed LNI 192.168.201.111@tcp [ 2943.138419] Key type .llcrypt unregistered [ 2943.140644] Key type ._llcrypt unregistered [ 2962.154383] Key type ._llcrypt registered [ 2962.158479] Key type .llcrypt registered [ 2962.284226] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2963.456780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2963.478407] alg: No test for adler32 (adler32-zlib) [ 2964.619456] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 2964.906944] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2966.601301] Key type lgssc registered [ 2967.756949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2975.753510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2975.771905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.244508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.129975] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2987.490981] LustreError: 99280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.512399] LustreError: 99280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2990.114385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.408565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2994.798641] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3000.887314] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3006.143245] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3012.695297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3013.133180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3019.139440] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3022.384723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3026.992175] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3036.641864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3036.667365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.677987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3037.669251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3037.670833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.119830] Lustre: server umount lustre-OST0000 complete [ 3046.885803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3046.899494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.918636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3046.927351] Lustre: Skipped 1 previous similar message [ 3052.429827] Lustre: server umount lustre-MDT0000 complete [ 3053.027899] LustreError: 100325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.048859] LustreError: 100325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3056.844749] LustreError: 100015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786434719 with bad export cookie 8792934331128386573 [ 3056.845618] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3056.860812] LustreError: 100015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3057.278071] Lustre: server umount lustre-MDT0001 complete [ 3069.819946] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3072.813706] Key type lgssc unregistered [ 3073.096200] LNet: 102813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3073.102750] LNetError: 102813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3073.119834] LNet: Removed LNI 192.168.201.111@tcp [ 3073.946171] Key type .llcrypt unregistered [ 3073.948858] Key type ._llcrypt unregistered [ 3090.865694] Key type ._llcrypt registered [ 3090.869076] Key type .llcrypt registered [ 3090.944416] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3091.913981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3092.135594] alg: No test for adler32 (adler32-zlib) [ 3093.275577] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 3093.532124] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3095.226802] Key type lgssc registered [ 3096.900312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3104.017528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3104.017846] LustreError: 103904:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3104.028080] Lustre: Skipped 1 previous similar message [ 3104.035734] LustreError: 103903:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3104.040661] LustreError: 103904:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3104.052846] LustreError: 103904:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3104.063516] LustreError: 103904:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3104.086823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.709301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.842371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.168356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3118.999915] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3125.093718] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing lsmod [ 3128.445078] LustreError: 105107:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3128.465215] LustreError: 105111:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3128.469804] LustreError: 105107:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 3128.483428] LustreError: 105107:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3128.490816] LustreError: 105107:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3128.504746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3128.939622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.989304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3134.945455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3134.951528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.974763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.464335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.485280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.491269] Lustre: Skipped 1 previous similar message [ 3135.512571] Lustre: Skipped 2 previous similar messages [ 3139.626905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.636452] Lustre: Skipped 1 previous similar message [ 3144.674908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3144.688942] Lustre: Skipped 2 previous similar messages [ 3146.208727] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3146.515335] Lustre: server umount lustre-MDT0000 complete [ 3149.808316] LustreError: 103942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.825095] LustreError: 103942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3149.968227] LustreError: 103918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786434812 with bad export cookie 3766146856969013993 [ 3149.977320] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3149.987404] LustreError: 103918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3151.968318] LustreError: 104372:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b4c29e24 x1873212609682944/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3152.142218] Lustre: server umount lustre-MDT0001 complete [ 3166.578249] Lustre: server umount lustre-OST0000 complete [ 3174.489854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.030041] LustreError: 106176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.143319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.280975] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3180.522419] LustreError: 106177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.634468] LustreError: 106176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.131395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.448167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3195.885487] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3205.388473] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3211.667313] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3219.597424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.926562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.983808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3227.541907] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3237.104787] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3245.538667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.543145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3245.552897] Lustre: Skipped 1 previous similar message [ 3245.575039] Lustre: Skipped 1 previous similar message [ 3249.909839] Lustre: server umount lustre-OST0000 complete [ 3255.778557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.780699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.794375] Lustre: Skipped 1 previous similar message [ 3260.211294] Lustre: server umount lustre-MDT0000 complete [ 3260.898817] LustreError: 106176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.925116] LustreError: 106176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3263.795205] LustreError: 106158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786434926 with bad export cookie 3766146856969014938 [ 3263.797505] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3263.807859] LustreError: 106158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3264.227658] Lustre: server umount lustre-MDT0001 complete [ 3277.115804] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3280.304623] Key type lgssc unregistered [ 3280.719227] LNet: 109712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3280.729628] LNetError: 109712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3280.773611] LNet: Removed LNI 192.168.201.111@tcp [ 3281.800941] Key type .llcrypt unregistered [ 3281.808229] Key type ._llcrypt unregistered [ 3305.282522] Key type ._llcrypt registered [ 3305.284478] Key type .llcrypt registered [ 3305.378913] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3306.469809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3306.586099] alg: No test for adler32 (adler32-zlib) [ 3307.761775] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 3308.174697] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3309.920245] Key type lgssc registered [ 3311.129235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3318.749707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3318.774530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.275873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.039346] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3330.535900] LustreError: 111129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.559046] LustreError: 111129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3335.655920] LustreError: 111130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.316234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.171853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3344.958858] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3352.885355] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3358.826991] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3365.029858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.340381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.386954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3371.107035] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3378.874958] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3388.108088] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3392.020912] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3398.127818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.132251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.167779] Lustre: Skipped 1 previous similar message [ 3398.191152] Lustre: Skipped 1 previous similar message [ 3403.233832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.252932] Lustre: Skipped 1 previous similar message [ 3403.441457] Lustre: server umount lustre-OST0000 complete [ 3408.865549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3408.879442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.892800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.495511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.499886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.540138] Lustre: Skipped 1 previous similar message [ 3415.167202] Lustre: server umount lustre-MDT0000 complete [ 3418.598201] LustreError: 112564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.620061] LustreError: 112564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3419.483359] LustreError: 111110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786435081 with bad export cookie 14188889227654200191 [ 3419.486195] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.495263] LustreError: 111110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3419.981516] Lustre: server umount lustre-MDT0001 complete [ 3432.433887] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3435.656175] Key type lgssc unregistered [ 3436.068404] LNet: 114777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3436.079495] LNetError: 114777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3436.114715] LNet: Removed LNI 192.168.201.111@tcp [ 3437.251879] Key type .llcrypt unregistered [ 3437.253494] Key type ._llcrypt unregistered [ 3458.904853] Key type ._llcrypt registered [ 3458.909508] Key type .llcrypt registered [ 3459.110188] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3460.915940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3460.942541] alg: No test for adler32 (adler32-zlib) [ 3462.005622] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 3462.267252] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3464.008185] Key type lgssc registered [ 3465.218342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3473.475558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3473.548509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.239354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.018843] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3485.669214] LustreError: 116000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3485.693837] LustreError: 116000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3490.135168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.519402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3495.320960] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3503.060510] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3509.036789] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3516.993422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.418730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.516270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3524.865490] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3533.815586] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3545.057404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3545.060720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.075531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3545.113301] Lustre: Skipped 1 previous similar message [ 3549.891337] Lustre: server umount lustre-OST0000 complete [ 3555.298199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.300501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3555.314758] Lustre: Skipped 1 previous similar message [ 3555.324088] Lustre: Skipped 1 previous similar message [ 3559.896461] Lustre: server umount lustre-MDT0000 complete [ 3560.421637] LustreError: 116001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.446562] LustreError: 116001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3563.415396] LustreError: 115982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786435225 with bad export cookie 12093504145635849826 [ 3563.456835] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3564.209848] Lustre: server umount lustre-MDT0001 complete [ 3576.599317] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3580.133248] Key type lgssc unregistered [ 3580.471409] LNet: 119536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3580.486438] LNetError: 119536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3580.510355] LNet: Removed LNI 192.168.201.111@tcp [ 3581.453329] Key type .llcrypt unregistered [ 3581.465913] Key type ._llcrypt unregistered [ 3611.236502] Key type ._llcrypt registered [ 3611.240163] Key type .llcrypt registered [ 3611.432581] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3613.461537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3613.491496] alg: No test for adler32 (adler32-zlib) [ 3614.812958] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 3615.131048] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3616.849933] Key type lgssc registered [ 3618.310803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3626.777715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3626.811853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.319027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.848970] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3638.760672] LustreError: 120972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3638.802196] LustreError: 120972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3642.510354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.046799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3648.540676] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3657.492701] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3664.227652] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3672.106085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3672.953420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.912051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3681.448191] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3690.438818] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3698.595940] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3704.186764] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3704.191528] Lustre: Skipped 2 previous similar messages [ 3707.936595] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3707.966427] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3709.707059] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3709.716967] Lustre: Skipped 2 previous similar messages [ 3717.662675] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3717.665773] Lustre: Skipped 1 previous similar message [ 3722.589399] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.11@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3722.595649] Lustre: Skipped 2 previous similar messages [ 3727.487742] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.11@tcp 192.168.0.[2,10]@tcp [ 3732.896725] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.11/24@tcp in log lustre-MDT0000 [ 3732.900550] Lustre: Skipped 2 previous similar messages [ 3737.151689] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.11/24@tcp [ 3737.166304] Lustre: Skipped 1 previous similar message [ 3744.741292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.746216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3744.758255] Lustre: Skipped 1 previous similar message [ 3744.766064] Lustre: Skipped 1 previous similar message [ 3749.864602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3749.874802] Lustre: Skipped 1 previous similar message [ 3750.252655] Lustre: server umount lustre-OST0000 complete [ 3754.980701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.989671] Lustre: Skipped 1 previous similar message [ 3754.992564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3754.998020] Lustre: Skipped 1 previous similar message [ 3760.099710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3760.117506] Lustre: Skipped 1 previous similar message [ 3760.931766] Lustre: server umount lustre-MDT0000 complete [ 3765.021591] LustreError: 123601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786435427 with bad export cookie 10254295789731228452 [ 3765.033210] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3765.038796] LustreError: 123601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3765.688628] Lustre: server umount lustre-MDT0001 complete [ 3777.495847] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3780.545532] Key type lgssc unregistered [ 3780.899828] LNet: 125918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3780.913723] LNetError: 125918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3780.935263] LNet: Removed LNI 192.168.201.111@tcp [ 3781.656553] Key type .llcrypt unregistered [ 3781.657702] Key type ._llcrypt unregistered [ 3811.415490] Key type ._llcrypt registered [ 3811.418273] Key type .llcrypt registered [ 3811.535726] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3813.013683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3813.123040] alg: No test for adler32 (adler32-zlib) [ 3814.349729] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 3814.605501] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3816.257622] Key type lgssc registered [ 3817.472532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3824.862631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3824.897612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3826.417522] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3826.446312] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.11/24@tcp [ 3826.453704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.137535] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3835.884146] LustreError: 127497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3835.900376] LustreError: 127497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3838.721655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.920858] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3838.957178] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.11/24@tcp [ 3839.007817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3843.423741] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3850.257192] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3856.177833] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3862.908942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3863.341344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3869.858609] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3871.558734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3876.638270] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.074131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3887.083908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.085958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.089017] Lustre: Skipped 1 previous similar message [ 3887.092891] Lustre: Skipped 1 previous similar message [ 3892.195160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.314258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.316226] Lustre: Skipped 1 previous similar message [ 3900.385451] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3900.528714] Lustre: server umount lustre-OST0000 complete [ 3905.505214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3905.513649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.538585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.554249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.624774] Lustre: server umount lustre-MDT0000 complete [ 3912.674159] LustreError: 127492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.691970] LustreError: 127492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3914.277323] LustreError: 127477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786435576 with bad export cookie 15407583086688453517 [ 3914.284044] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3914.294833] LustreError: 127477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3914.580852] Lustre: server umount lustre-MDT0001 complete [ 3923.760959] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3926.071530] Key type lgssc unregistered [ 3926.338731] LNet: 131089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3926.354483] LNetError: 131089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3926.374285] LNet: Removed LNI 192.168.201.111@tcp [ 3927.196970] Key type .llcrypt unregistered [ 3927.199298] Key type ._llcrypt unregistered [ 3959.419385] Key type ._llcrypt registered [ 3959.428486] Key type .llcrypt registered [ 3959.580497] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 3977.791573] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3979.567456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3979.875933] alg: No test for adler32 (adler32-zlib) [ 3981.188798] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 3981.577794] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3983.424207] Key type lgssc registered [ 3984.884908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4020.903666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4020.940523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.163063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4022.208346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4022.273622] Lustre: lustre-MDT0000: new disk, initializing [ 4022.313826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.323186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4027.597385] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4038.659939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.857479] Lustre: 134497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4038.978605] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4038.984240] Lustre: Skipped 1 previous similar message [ 4039.094576] Lustre: lustre-MDT0001: new disk, initializing [ 4039.228539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4039.296318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4039.306833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4043.846805] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4052.738354] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4058.337940] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4066.162550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4066.500446] Lustre: lustre-OST0000: new disk, initializing [ 4066.504415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4066.521347] Lustre: 136465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4066.672836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4074.405145] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4074.679521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4074.694637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4074.897287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4085.254177] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4090.650578] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4090.887134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4096.158830] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4096.398362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4100.578562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4100.599829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.611070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4100.613016] Lustre: Skipped 1 previous similar message [ 4100.635656] Lustre: Skipped 1 previous similar message [ 4104.627771] Lustre: server umount lustre-OST0000 complete [ 4110.819086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.830478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4110.835612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.844820] Lustre: Skipped 1 previous similar message [ 4110.869121] LustreError: Skipped 1 previous similar message [ 4115.492893] Lustre: server umount lustre-MDT0000 complete [ 4120.088464] LustreError: 134487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786435782 with bad export cookie 1334410155131332096 [ 4120.092314] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4120.099696] LustreError: 134487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4120.767832] Lustre: server umount lustre-MDT0001 complete [ 4132.075700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.598721] LustreError: 138683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.627094] LustreError: 138683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4132.701103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4137.791587] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4137.968074] LustreError: 138684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.053425] LustreError: 138683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.170779] LustreError: 138684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.800637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.370628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4154.219205] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4162.376592] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4169.354316] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4174.944892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4175.212368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.981345] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4190.614531] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4195.974798] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4196.226821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4201.204027] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4201.502508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4210.750659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4210.914143] Lustre: lustre-OST0001: new disk, initializing [ 4210.916884] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4210.926697] Lustre: 142054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4211.118348] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4218.032884] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4220.015064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4220.025662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4220.134535] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4226.316547] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4226.626547] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4231.183321] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4231.398705] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4237.769146] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91719044d000.ost_server_uuid 50 [ 4239.296185] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91719044d000.ost_server_uuid in FULL state after 0 sec [ 4251.115877] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4251.117939] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.120151] LustreError: Skipped 1 previous similar message [ 4251.125613] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4251.127784] Lustre: Skipped 1 previous similar message [ 4251.141648] Lustre: Skipped 2 previous similar messages [ 4256.225478] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4261.344476] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4261.354096] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4261.371276] Lustre: Skipped 1 previous similar message [ 4261.655253] Lustre: server umount lustre-OST0001 complete [ 4266.471432] LustreError: 140506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.508035] LustreError: 140506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4269.538082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4269.544846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.561634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4273.556293] Lustre: server umount lustre-OST0000 complete [ 4281.319270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4281.331962] LustreError: Skipped 1 previous similar message [ 4281.338733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.358481] Lustre: Skipped 1 previous similar message [ 4284.859906] Lustre: server umount lustre-MDT0000 complete [ 4286.947808] LustreError: 138683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4286.973666] LustreError: 138683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4288.657635] LustreError: 142053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786435950 with bad export cookie 1334410155131332978 [ 4288.670705] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4289.161639] Lustre: server umount lustre-MDT0001 complete [ 4309.541316] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 4312.310383] Key type lgssc unregistered [ 4312.582029] LNet: 145064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4312.589413] LNetError: 145064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4313.639086] LNet: Removed LNI 192.168.201.111@tcp [ 4314.596763] Key type .llcrypt unregistered [ 4314.598857] Key type ._llcrypt unregistered [ 4356.516622] Key type ._llcrypt registered [ 4356.518839] Key type .llcrypt registered [ 4356.631664] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 4371.721233] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 4373.189563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4373.236891] alg: No test for adler32 (adler32-zlib) [ 4374.273372] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 4374.627794] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 4376.344928] Key type lgssc registered [ 4377.818062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4409.956267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4409.982972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.293406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4411.325729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4411.425936] Lustre: lustre-MDT0000: new disk, initializing [ 4411.500875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.519717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4416.376352] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4427.059251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.214886] Lustre: 149226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4427.287072] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4427.289309] Lustre: Skipped 1 previous similar message [ 4427.380956] Lustre: lustre-MDT0001: new disk, initializing [ 4427.503177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4427.562842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4427.580258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4432.302306] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4442.035257] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4448.205689] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4455.873299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4456.240880] Lustre: lustre-OST0000: new disk, initializing [ 4456.245798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4456.256516] Lustre: 151193:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4456.364822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.658197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4461.685970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4461.832577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4462.912070] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4471.931402] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4475.644812] Lustre: Failing over lustre-OST0000 [ 4475.857728] Lustre: server umount lustre-OST0000 complete [ 4492.779816] Lustre: 147452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786436139/real 1786436139] req@00000000534cfd93 x1873213952918528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786436155 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4492.834529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.792152] Lustre: 147449:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786436139/real 1786436139] req@00000000fc3a0384 x1873213952918272/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786436155 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4493.793186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.822903] Lustre: 147449:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4495.133144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4495.584737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.607985] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4498.656144] Lustre: 147451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786436144/real 1786436144] req@00000000d8a5bb8a x1873213952919680/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786436160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4498.674901] Lustre: 147451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4502.130747] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4502.826297] Lustre: 147451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786436149/real 1786436149] req@000000002404d80d x1873213952920576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786436165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4502.872187] Lustre: 147451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4503.721441] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4505.183767] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4505.184523] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4505.390428] Lustre: Failing over lustre-MDT0000 [ 4505.656961] Lustre: server umount lustre-MDT0000 complete [ 4508.448337] Lustre: 147451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786436154/real 1786436154] req@00000000e655cdd3 x1873213952921600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786436170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4508.478245] Lustre: 147451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4508.839756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.857509] Lustre: Skipped 1 previous similar message [ 4514.275321] LustreError: 149232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.296625] LustreError: 149232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4515.394557] LustreError: 149234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.395221] LustreError: 151202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.415293] LustreError: 151202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4523.607418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.692651] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4523.890208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.941313] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4525.632653] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4528.531753] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4529.142420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4529.147231] Lustre: Skipped 1 previous similar message [ 4529.156544] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4529.219954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4536.291508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4536.296035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.313346] Lustre: Skipped 1 previous similar message [ 4536.318958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4539.369924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4542.467513] Lustre: server umount lustre-OST0000 complete [ 4549.606115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.615173] Lustre: Skipped 2 previous similar messages [ 4549.616321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4549.620435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4549.631570] Lustre: Skipped 2 previous similar messages [ 4552.101534] Lustre: server umount lustre-MDT0000 complete [ 4554.727277] LustreError: 149238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.747273] LustreError: 149238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4555.482769] LustreError: 149219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786436217 with bad export cookie 16008133275774143853 [ 4555.485799] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4555.490444] LustreError: 149219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4555.738586] Lustre: server umount lustre-MDT0001 complete [ 4566.076971] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 4569.258555] Key type lgssc unregistered [ 4569.561347] LNet: 154845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4569.566892] LNetError: 154845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4570.609498] LNet: Removed LNI 192.168.201.111@tcp [ 4571.555401] Key type .llcrypt unregistered [ 4571.557784] Key type ._llcrypt unregistered [ 4610.754581] Key type ._llcrypt registered [ 4610.756238] Key type .llcrypt registered [ 4610.839942] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 4611.526216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4611.546475] alg: No test for adler32 (adler32-zlib) [ 4612.931772] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 4613.176302] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 4614.897334] Key type lgssc registered [ 4615.860802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4622.957849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4622.979546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.642311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.183941] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4634.090387] LustreError: 156364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4634.108437] LustreError: 156364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4638.906427] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4639.203389] LustreError: 156365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.778263] LustreError: 156393:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4642.804727] LustreError: 156393:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4642.813450] LustreError: 156393:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4645.222103] Lustre: server umount lustre-MDT0000 complete [ 4660.113818] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 4669.522341] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 4705.473686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.743362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4705.789304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4705.897411] Lustre: lustre-MDT0000: new disk, initializing [ 4706.030677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.076703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4710.720363] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4721.446319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.590563] Lustre: 159701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4721.705966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4721.718247] Lustre: Skipped 1 previous similar message [ 4721.815293] Lustre: lustre-MDT0001: new disk, initializing [ 4721.948960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4721.991340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4721.997054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4726.954532] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4737.109461] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4742.963417] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4749.524639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4749.824429] Lustre: lustre-OST0000: new disk, initializing [ 4749.828534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4749.835800] Lustre: 161669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4749.920552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.476882] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4758.582945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4758.599051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4758.691361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4764.401162] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4768.491815] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4770.439939] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:20:30 (1786436430) [ 4978.144738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4978.150427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4978.157369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4979.171820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4979.172542] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4979.228856] Lustre: Skipped 1 previous similar message [ 4983.073552] Lustre: server umount lustre-MDT0000 complete [ 4984.292485] LustreError: 160893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.335652] LustreError: 160893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4988.022894] LustreError: 159693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786436650 with bad export cookie 728751028385203131 [ 4988.033581] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4988.040741] LustreError: 159693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4989.409401] LustreError: 159708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4989.441111] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.459276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4989.464152] Lustre: Skipped 2 previous similar messages [ 4993.443034] LustreError: 161676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4993.444902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4994.448125] Lustre: server umount lustre-MDT0001 complete [ 5005.237882] Lustre: server umount lustre-OST0000 complete [ 5016.504695] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 5028.570240] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 5067.200552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5067.579964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5067.608318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5067.730878] Lustre: lustre-MDT0000: new disk, initializing [ 5067.884422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5067.952877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5073.156056] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5083.912673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5084.062316] Lustre: 166341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5084.120071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5084.126599] Lustre: Skipped 1 previous similar message [ 5084.213939] Lustre: lustre-MDT0001: new disk, initializing [ 5084.321985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5084.406599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5084.438387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5089.650677] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5099.887260] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5106.509585] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5113.343812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5113.713965] Lustre: lustre-OST0000: new disk, initializing [ 5113.717797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5113.724975] Lustre: 168308:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5113.892234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.597160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5120.614062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5120.714073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5120.809472] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5130.554651] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5135.565786] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5135.758420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5140.951952] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5141.209510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5146.594183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5146.594616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.616232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5149.378879] Lustre: server umount lustre-OST0000 complete [ 5156.325706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5156.332613] LustreError: Skipped 1 previous similar message [ 5156.337562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.350165] Lustre: Skipped 1 previous similar message [ 5156.355337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.360449] Lustre: Skipped 1 previous similar message [ 5161.382641] Lustre: server umount lustre-MDT0000 complete [ 5161.965312] LustreError: 166353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.005132] LustreError: 166353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5165.146224] LustreError: 166335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786436827 with bad export cookie 728751028385361961 [ 5165.147067] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5165.166388] LustreError: 166335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5165.589964] Lustre: server umount lustre-MDT0001 complete [ 5174.804355] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:27:15 (1786436835) [ 5192.939760] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 5204.593908] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 5241.912933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5242.192377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5242.239700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5242.342790] Lustre: lustre-MDT0000: new disk, initializing [ 5242.446215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.468922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5247.106549] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5257.347930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.518210] Lustre: 173064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5257.569219] LustreError: 173077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5257.779550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5262.504103] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5271.318277] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5277.814958] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5284.706334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5284.953340] Lustre: lustre-OST0000: new disk, initializing [ 5284.956119] Lustre: Skipped 1 previous similar message [ 5284.959274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5284.962493] Lustre: Skipped 2 previous similar messages [ 5284.966669] Lustre: 175043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5291.354898] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5293.628291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5293.640307] Lustre: Skipped 1 previous similar message [ 5293.650172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5293.799412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5302.467786] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5314.530626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5314.540818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.560602] Lustre: Skipped 1 previous similar message [ 5314.564266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5314.566012] Lustre: Skipped 1 previous similar message [ 5317.675314] Lustre: server umount lustre-OST0000 complete [ 5324.257624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5324.260827] LustreError: Skipped 1 previous similar message [ 5329.829065] Lustre: server umount lustre-MDT0000 complete [ 5329.889467] LustreError: 175341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5329.918478] LustreError: 175341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5333.726291] LustreError: 173058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786436995 with bad export cookie 728751028385362843 [ 5333.745154] LustreError: 173058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5333.748638] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5334.252816] Lustre: server umount lustre-MDT0001 complete [ 5342.857412] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:30:03 (1786437003) [ 5360.438873] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 5370.669612] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 5407.127799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.508881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5407.545502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5407.655157] Lustre: lustre-MDT0000: new disk, initializing [ 5407.807936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.811023] Lustre: Skipped 2 previous similar messages [ 5407.846363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5414.377601] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5424.864276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.005773] Lustre: 179372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5425.228049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5429.760611] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5439.295497] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5445.110909] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5451.907304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5452.302831] Lustre: 181347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5459.553605] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5460.652095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5460.807382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5468.236235] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5476.323102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.331095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5476.340562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5476.340571] Lustre: Skipped 2 previous similar messages [ 5476.354027] Lustre: Skipped 4 previous similar messages [ 5476.384522] LustreError: Skipped 1 previous similar message [ 5481.243991] Lustre: server umount lustre-OST0000 complete [ 5491.563406] Lustre: server umount lustre-MDT0000 complete [ 5491.699896] LustreError: 179382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.722978] LustreError: 179382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5496.078472] LustreError: 179364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786437158 with bad export cookie 728751028385363928 [ 5496.096861] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5496.708675] Lustre: server umount lustre-MDT0001 complete [ 5508.240238] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 5511.996572] Key type lgssc unregistered [ 5512.484642] LNet: 183205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5512.500871] LNetError: 183205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5513.581106] LNet: Removed LNI 192.168.201.111@tcp [ 5514.547794] Key type .llcrypt unregistered [ 5514.550350] Key type ._llcrypt unregistered [ 5539.094880] Key type ._llcrypt registered [ 5539.098264] Key type .llcrypt registered [ 5539.219209] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 5540.583070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5540.708041] alg: No test for adler32 (adler32-zlib) [ 5541.745172] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 5541.940523] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 5543.616366] Key type lgssc registered [ 5545.072779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5553.307471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5553.329767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5554.988121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.987437] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5565.413385] LustreError: 184622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5565.429212] LustreError: 184622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5568.228864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.579461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5573.020762] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5580.499657] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5586.455558] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5593.612942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5593.999531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5600.191300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5600.275900] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5607.752528] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5628.902542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5628.903790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.956745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5633.434432] Lustre: server umount lustre-OST0000 complete [ 5639.138980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.158501] Lustre: Skipped 1 previous similar message [ 5639.173776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5639.194351] Lustre: Skipped 2 previous similar messages [ 5643.573995] Lustre: server umount lustre-MDT0000 complete [ 5644.259582] LustreError: 184622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5644.287272] LustreError: 184622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5647.737339] LustreError: 184601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786437309 with bad export cookie 4003989650103393811 [ 5647.743475] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5647.753020] LustreError: 184601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5648.104829] Lustre: server umount lustre-MDT0001 complete [ 5660.898494] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 5664.325764] Key type lgssc unregistered [ 5664.760856] LNet: 188180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5664.783418] LNetError: 188180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5664.832331] LNet: Removed LNI 192.168.201.111@tcp [ 5665.823893] Key type .llcrypt unregistered [ 5665.832539] Key type ._llcrypt unregistered [ 5692.515781] Key type ._llcrypt registered [ 5692.518282] Key type .llcrypt registered [ 5692.602947] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 5694.597588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5694.619511] alg: No test for adler32 (adler32-zlib) [ 5695.962813] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 5696.509077] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 5698.321284] Key type lgssc registered [ 5700.563650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5710.875641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5710.886576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5712.202334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5717.393593] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5722.602704] LustreError: 189601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.639035] LustreError: 189601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5725.653967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.102605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5730.795475] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5737.815143] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5743.258472] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5751.453664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5751.724247] LustreError: 191420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.939402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5757.072913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5758.974718] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5769.280608] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5775.967106] Lustre: lustre-OST0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 5777.891769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5777.899701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5777.907735] Lustre: Skipped 1 previous similar message [ 5777.926490] Lustre: Skipped 1 previous similar message [ 5781.084473] Lustre: lustre-OST0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 5781.950932] Lustre: server umount lustre-OST0000 complete [ 5789.617574] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff917189cc2800.ost_server_uuid 50 [ 5791.194828] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5808.100683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5808.105342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.122156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5812.399637] Lustre: server umount lustre-MDT0000 complete [ 5813.740235] LustreError: 190841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5813.763142] LustreError: 190841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5816.546563] LustreError: 189582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786437478 with bad export cookie 16435650262450453037 [ 5816.547468] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5816.564321] LustreError: 189582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5817.211876] Lustre: server umount lustre-MDT0001 complete [ 5827.342215] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:38:07 (1786437487) [ 5833.756902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5834.231175] LustreError: 193516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.376196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5839.641682] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5839.843661] LustreError: 193517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.898815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5848.180372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5852.727046] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5862.855196] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5870.250494] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5878.376167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.270714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5880.331383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5887.986245] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5896.517221] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5903.786105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5904.010657] Lustre: lustre-OST0001: new disk, initializing [ 5904.013533] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5904.018508] Lustre: 196358:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5904.104110] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5911.088809] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5913.195034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5913.211375] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5913.302202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5921.021922] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5923.820737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5923.828859] Lustre: Skipped 1 previous similar message [ 5923.833369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5923.836517] Lustre: Skipped 2 previous similar messages [ 5932.105128] Lustre: lustre-OST0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 5932.114532] Lustre: Skipped 3 previous similar messages [ 5938.148315] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5938.219271] Lustre: server umount lustre-OST0000 complete [ 5939.173819] LustreError: 196821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5939.202590] LustreError: 196821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5944.610659] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5944.911248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5949.410770] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5949.593645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5957.698528] LustreError: 195338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.713780] LustreError: 195338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5964.777011] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.779100] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5964.787147] Lustre: Skipped 2 previous similar messages [ 5964.787680] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5964.787685] Lustre: Skipped 3 previous similar messages [ 5964.833833] LustreError: Skipped 1 previous similar message [ 5969.612552] Lustre: server umount lustre-OST0001 complete [ 5975.014983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.041982] Lustre: Skipped 1 previous similar message [ 5979.855150] Lustre: server umount lustre-MDT0000 complete [ 5983.824149] LustreError: 193497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786437646 with bad export cookie 16435650262450454108 [ 5983.835208] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5983.844326] LustreError: 193497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5984.185590] Lustre: server umount lustre-MDT0001 complete [ 6010.836658] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:41:11 (1786437671) [ 6018.084517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6018.399275] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6018.434702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6018.670512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6022.938950] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6030.854032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.063815] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6031.118268] Lustre: 199808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6036.175136] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6043.386924] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6049.751499] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6056.235637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6056.454424] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6062.642873] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6066.367051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6071.018576] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6077.607089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6077.746543] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6083.650428] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6091.707636] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6096.394748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6096.417660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6096.428180] Lustre: Skipped 3 previous similar messages [ 6101.376052] Lustre: server umount lustre-OST0000 complete [ 6101.489187] LustreError: 201659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6101.524793] LustreError: 201659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6121.446280] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6121.449718] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.454552] Lustre: Skipped 1 previous similar message [ 6123.901183] Lustre: server umount lustre-OST0001 complete [ 6128.609643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6128.614354] LustreError: Skipped 1 previous similar message [ 6135.038357] Lustre: server umount lustre-MDT0000 complete [ 6139.277464] LustreError: 200565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786437801 with bad export cookie 16435650262450455515 [ 6139.280754] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6139.288174] LustreError: 200565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6139.606955] Lustre: server umount lustre-MDT0001 complete [ 6163.588725] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:43:44 (1786437824) [ 6180.456500] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 6192.354935] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 6231.416451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6231.729526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6231.784445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6231.937193] Lustre: lustre-MDT0000: new disk, initializing [ 6232.067311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.072893] Lustre: Skipped 3 previous similar messages [ 6232.087388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6237.234529] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6247.805792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6247.911079] Lustre: 207906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6247.995215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6247.998941] Lustre: Skipped 1 previous similar message [ 6248.056523] Lustre: lustre-MDT0001: new disk, initializing [ 6248.162252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6248.167285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6253.571101] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6264.669919] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6270.960124] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6277.982552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6278.445458] Lustre: lustre-OST0000: new disk, initializing [ 6278.459398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6278.477577] Lustre: 209875:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6284.602451] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6286.945365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6286.954601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6287.078139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6293.723740] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6298.158709] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6298.348464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6302.795488] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6303.028361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6307.809619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6307.809866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.832263] LustreError: Skipped 1 previous similar message [ 6307.840576] Lustre: Skipped 3 previous similar messages [ 6307.842584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6307.858328] Lustre: Skipped 7 previous similar messages [ 6319.072725] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6319.286245] Lustre: server umount lustre-OST0000 complete [ 6329.562980] Lustre: server umount lustre-MDT0000 complete [ 6333.104716] LustreError: 207898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786437995 with bad export cookie 16435650262450457076 [ 6333.116323] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.124058] LustreError: 207898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6333.534744] Lustre: server umount lustre-MDT0001 complete [ 6343.361920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.937213] LustreError: 212092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6343.962783] LustreError: 212092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6350.516287] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6359.769542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.528938] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6373.325611] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6379.572228] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6386.492223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6386.898292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6386.903798] Lustre: Skipped 4 previous similar messages [ 6391.775488] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6398.142202] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6404.142425] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6404.466975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6410.362729] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6410.719051] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6414.820138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.823262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.852238] Lustre: Skipped 3 previous similar messages [ 6419.454631] Lustre: server umount lustre-OST0000 complete [ 6426.880643] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6431.180575] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6459.873867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6466.350819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.365432] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6466.856413] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6474.270560] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6478.273285] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6481.762460] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.784188] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6487.038409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6487.051473] LustreError: Skipped 1 previous similar message [ 6487.060031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6487.076330] Lustre: Skipped 12 previous similar messages [ 6488.433393] Lustre: server umount lustre-OST0000 complete [ 6498.738046] Lustre: server umount lustre-MDT0000 complete [ 6502.370111] LustreError: 212093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6502.390762] LustreError: 212093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6502.878834] LustreError: 212073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786438165 with bad export cookie 16435650262450457958 [ 6502.895126] LustreError: 212073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6502.895236] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6503.322796] Lustre: server umount lustre-MDT0001 complete [ 6511.666394] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:49:32 (1786438172) [ 6518.297924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6524.154817] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6533.451337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.635708] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6546.394412] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6552.986617] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6560.588036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6567.838746] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6575.927855] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6581.021499] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6581.279556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6586.225303] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6586.467436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6593.868408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6594.084770] Lustre: lustre-OST0001: new disk, initializing [ 6594.089254] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6594.097909] Lustre: 221431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6601.712957] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6603.352758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6603.363454] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6603.475221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6611.829787] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6616.785836] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6616.988825] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6621.502616] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6621.739135] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6624.240141] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.241798] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6624.249661] Lustre: Skipped 5 previous similar messages [ 6624.289511] LustreError: Skipped 2 previous similar messages [ 6629.738546] Lustre: server umount lustre-OST0001 complete [ 6635.607345] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6635.819393] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6640.687497] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6640.953674] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6669.906609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6670.212590] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6670.218114] Lustre: Skipped 5 previous similar messages [ 6675.456956] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6675.478447] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6675.487462] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6675.506568] Lustre: Skipped 1 previous similar message [ 6677.136464] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6683.812886] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6685.026178] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6690.794827] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6690.811466] LustreError: Skipped 1 previous similar message [ 6692.842327] Lustre: server umount lustre-OST0001 complete [ 6714.742060] Lustre: server umount lustre-OST0000 complete [ 6724.801038] Lustre: server umount lustre-MDT0000 complete [ 6728.694692] LustreError: 218040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786438390 with bad export cookie 16435650262450458924 [ 6728.701135] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6728.707474] LustreError: 218040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6735.346270] Lustre: server umount lustre-MDT0001 complete [ 6759.671458] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:53:40 (1786438420) [ 6765.995836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6766.381716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6766.409027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6771.917407] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6782.644283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.840019] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6782.902750] Lustre: 226673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6788.715401] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6798.510753] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6804.432784] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6810.913121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6811.085704] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6816.709881] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6823.471276] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6831.847536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6831.914263] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6837.765505] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6847.666633] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6849.985758] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6856.801713] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6857.330071] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6864.961426] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6865.237813] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6871.253556] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91719153d000.ost_server_uuid 50 [ 6872.832656] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91719153d000.ost_server_uuid in IDLE state after 0 sec [ 6873.840803] Lustre: Permanently deactivating lustre-OST0001 [ 6873.869635] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6877.995605] Lustre: Permanently reactivating lustre-OST0001 [ 6878.045113] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6878.050105] Lustre: Skipped 2 previous similar messages [ 6883.814015] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6883.827974] LustreError: Skipped 1 previous similar message [ 6883.831915] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.848463] Lustre: Skipped 7 previous similar messages [ 6883.862179] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6883.870824] Lustre: Skipped 15 previous similar messages [ 6888.274804] Lustre: server umount lustre-OST0001 complete [ 6889.966278] LustreError: 228521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6889.979433] LustreError: 228521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 35 previous similar messages [ 6897.838938] Lustre: server umount lustre-OST0000 complete [ 6908.503486] Lustre: server umount lustre-MDT0000 complete [ 6912.236254] LustreError: 226666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786438574 with bad export cookie 16435650262450460212 [ 6912.241854] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6912.254055] LustreError: 226666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6912.558753] Lustre: server umount lustre-MDT0001 complete [ 6941.941215] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:56:41 (1786438601) [ 6952.807645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.319808] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6953.381988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6953.392113] Lustre: Skipped 2 previous similar messages [ 6957.147888] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6966.346666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.548907] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6966.593636] Lustre: 233369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6973.527170] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6981.924843] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6989.332619] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6995.502493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6995.686423] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6995.705621] Lustre: Permanently deactivating lustre-OST0000 [ 6995.718857] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7003.185888] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7004.206600] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7011.106796] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7018.004711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.078909] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7025.229965] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7027.354335] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7033.468029] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7035.311455] Lustre: Permanently reactivating lustre-OST0000 [ 7035.329158] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7035.332725] Lustre: Skipped 2 previous similar messages [ 7044.728658] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7044.747857] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 0000000095f4de09, cur 1786438706 deadline 1786438696 last 1786438666 [ 7044.846594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7057.383087] Lustre: server umount lustre-OST0001 complete [ 7075.303471] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7075.678185] Lustre: server umount lustre-OST0000 complete [ 7087.082858] Lustre: server umount lustre-MDT0000 complete [ 7091.455145] LustreError: 233361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786438753 with bad export cookie 16435650262450461850 [ 7091.467288] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7091.487456] LustreError: 233361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7092.064541] Lustre: server umount lustre-MDT0001 complete [ 7102.387594] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 7104.936980] Key type lgssc unregistered [ 7105.171261] LNet: 238252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7105.179980] LNetError: 238252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7105.198142] LNet: Removed LNI 192.168.201.111@tcp [ 7106.010529] Key type .llcrypt unregistered [ 7106.017935] Key type ._llcrypt unregistered [ 7127.647727] Key type ._llcrypt registered [ 7127.650657] Key type .llcrypt registered [ 7127.732671] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7128.810454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7128.852490] alg: No test for adler32 (adler32-zlib) [ 7129.924331] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 7130.200331] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 7131.864141] Key type lgssc registered [ 7132.819595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7142.505131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7142.523348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.990889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7148.406303] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7154.149822] LustreError: 239715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.172246] LustreError: 239715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7157.406931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.559737] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7157.594426] Lustre: 239703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7157.605724] Lustre: Permanently deactivating lustre-MDT0001 [ 7157.631482] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7157.910632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7162.652481] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7163.936260] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7170.817546] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7176.934636] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7183.843406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7184.152076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7189.300688] LustreError: 241559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.314475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7190.340669] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7194.601844] LustreError: 241558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.112397] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7199.733557] LustreError: 241861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.817736] LustreError: 241557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.832726] LustreError: 241557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7206.863808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7207.205737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7212.544151] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7212.563620] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7215.810342] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7223.470836] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7226.631499] Lustre: Permanently reactivating lustre-MDT0001 [ 7226.673936] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7226.680640] Lustre: Skipped 2 previous similar messages [ 7230.702837] Lustre: lustre-MDT0001: Client dd8a8d58-2ecb-4147-a3c9-44b1e1836383 (at 192.168.201.11@tcp) reconnecting [ 7235.199423] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7235.211971] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7235.220600] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7235.231305] LustreError: 243439:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7235.234765] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7235.243236] LustreError: 243439:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7235.243365] LustreError: 243439:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.111@tcp: cfg command failed: rc = -108 [ 7235.243390] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7235.243390] [ 7235.248242] LustreError: 239707:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7236.868868] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7238.352700] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7244.133455] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7244.395879] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7248.960899] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7249.209393] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7251.486512] Lustre: Permanently deactivating lustre-MDT0001 [ 7251.499436] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7251.503454] Lustre: Skipped 2 previous similar messages [ 7261.317703] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7271.910478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.916439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.945207] Lustre: Skipped 1 previous similar message [ 7271.967906] Lustre: Skipped 3 previous similar messages [ 7277.037360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7277.045026] Lustre: Skipped 3 previous similar messages [ 7277.165499] Lustre: server umount lustre-MDT0000 complete [ 7282.152322] LustreError: 239715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.165791] LustreError: 239715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7282.478138] LustreError: 244213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786438944 with bad export cookie 8782692031784061199 [ 7282.486086] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7282.486186] LustreError: 244213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7282.840260] Lustre: server umount lustre-MDT0001 complete [ 7288.526559] Lustre: server umount lustre-OST0000 complete [ 7299.321347] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:02:40 (1786438960) [ 7311.986610] Lustre: server umount lustre-OST0001 complete [ 7321.121291] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 7329.648735] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7366.851385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.235323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7367.267275] Lustre: Skipped 2 previous similar messages [ 7367.304716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7367.398493] Lustre: lustre-MDT0000: new disk, initializing [ 7367.533901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.553593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7373.436395] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7386.791403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7386.954844] Lustre: 248037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7387.013639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7387.017860] Lustre: Skipped 1 previous similar message [ 7387.112392] Lustre: lustre-MDT0001: new disk, initializing [ 7387.258074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7387.301621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7387.311675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7392.547321] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7404.073084] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7412.216185] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7418.775572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7419.114513] Lustre: lustre-OST0000: new disk, initializing [ 7419.122738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7419.141131] Lustre: 250007:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7419.280186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7425.610782] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7427.199472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7427.208178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7427.356795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7434.032234] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7436.250643] LustreError: 248043:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7445.083660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.228645] Lustre: lustre-OST0001: new disk, initializing [ 7445.231399] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7445.235416] Lustre: 251120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7446.297263] LustreError: 248043:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7446.349315] LustreError: 250015:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7451.042665] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7453.908788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7453.927292] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7454.116889] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7456.433063] LustreError: 250015:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7460.822498] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7462.542970] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7467.109197] Lustre: lustre-OST0001: Not available for connect from 192.168.201.11@tcp (stopping) [ 7469.539134] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7469.548938] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7469.553851] LustreError: Skipped 1 previous similar message [ 7469.558689] Lustre: Skipped 2 previous similar messages [ 7469.562620] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7469.577165] Lustre: Skipped 1 previous similar message [ 7474.678790] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7474.682697] Lustre: Skipped 2 previous similar messages [ 7479.267824] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7479.593885] Lustre: server umount lustre-OST0001 complete [ 7479.812357] LustreError: 250457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7479.850541] LustreError: 250457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7500.283967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7500.306828] Lustre: Skipped 1 previous similar message [ 7500.323614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7500.329612] Lustre: Skipped 1 previous similar message [ 7505.687190] Lustre: server umount lustre-OST0000 complete [ 7515.108862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7515.123200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.155775] Lustre: Skipped 1 previous similar message [ 7520.741931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7520.756027] Lustre: Skipped 6 previous similar messages [ 7526.368480] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7527.187804] Lustre: server umount lustre-MDT0000 complete [ 7530.276043] LustreError: 248042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7530.297126] LustreError: 248042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7531.932746] LustreError: 248027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786439194 with bad export cookie 8782692031784063796 [ 7531.940018] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7531.947964] LustreError: 248027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7546.336815] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7546.856884] Lustre: server umount lustre-MDT0001 complete [ 7560.913421] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 7564.349520] Key type lgssc unregistered [ 7564.795628] LNet: 253164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7564.805790] LNetError: 253164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7564.832830] LNet: Removed LNI 192.168.201.111@tcp [ 7565.969557] Key type .llcrypt unregistered [ 7565.971630] Key type ._llcrypt unregistered [ 7609.138413] Key type ._llcrypt registered [ 7609.146057] Key type .llcrypt registered [ 7609.307210] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7610.483322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7610.513504] alg: No test for adler32 (adler32-zlib) [ 7611.499144] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 7611.800543] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 7613.512221] Key type lgssc registered [ 7614.848975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7625.298146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7625.329472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.927716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7627.002345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7627.331679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7633.329789] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7643.342432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7643.417622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7643.441075] Lustre: 255379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7643.665755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7648.639670] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7656.379480] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7662.341548] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7670.244732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7670.452320] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7670.612624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.941713] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7677.125517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7684.925590] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7738.346960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7738.349421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7738.359161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7738.854945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7738.869443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7738.881785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7740.898708] Lustre: server umount lustre-OST0000 complete [ 7745.366879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7759.883514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7760.235950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.930251] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7776.132888] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7779.633116] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7779.650384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7779.653059] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7779.679545] Lustre: Skipped 1 previous similar message [ 7820.775083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7820.780032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.784584] LustreError: Skipped 1 previous similar message [ 7820.794949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7820.808749] Lustre: Skipped 1 previous similar message [ 7820.823558] Lustre: Skipped 1 previous similar message [ 7822.560378] Lustre: server umount lustre-OST0000 complete [ 7827.938730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7827.950184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.960751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7832.888177] Lustre: server umount lustre-MDT0000 complete [ 7836.612757] LustreError: 255370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786439498 with bad export cookie 6373585808167127178 [ 7836.628205] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7836.630451] LustreError: 255370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7837.058695] Lustre: server umount lustre-MDT0001 complete [ 7847.992757] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 7850.838837] Key type lgssc unregistered [ 7851.150280] LNet: 260803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7851.158369] LNetError: 260803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7851.183962] LNet: Removed LNI 192.168.201.111@tcp [ 7852.088183] Key type .llcrypt unregistered [ 7852.092124] Key type ._llcrypt unregistered [ 7878.359936] Key type ._llcrypt registered [ 7878.375470] Key type .llcrypt registered [ 7878.549448] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7879.813089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7880.131872] alg: No test for adler32 (adler32-zlib) [ 7881.333325] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 7881.734613] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 7883.440330] Key type lgssc registered [ 7884.588194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7893.705551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7893.726590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7895.249340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7900.672569] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7905.763974] LustreError: 262221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7905.773698] LustreError: 262221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7910.881880] LustreError: 262220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7910.946028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.448115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7917.026559] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7924.401785] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7930.207040] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7938.088069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7938.669277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7940.736272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7946.129571] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7954.309612] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7987.177848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7987.181121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7987.185611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7987.198966] Lustre: Skipped 1 previous similar message [ 7988.963083] Lustre: server umount lustre-OST0000 complete [ 7993.315395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7993.325664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7993.338691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7993.345122] Lustre: Skipped 1 previous similar message [ 7997.430671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7997.434738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.601409] Lustre: server umount lustre-MDT0000 complete [ 8002.530066] LustreError: 262217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8002.546534] LustreError: 262217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8003.761063] LustreError: 262201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786439665 with bad export cookie 14792046460742258177 [ 8003.772062] LustreError: 262201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8003.774405] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8004.242884] Lustre: server umount lustre-MDT0001 complete [ 8015.515581] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8018.873278] Key type lgssc unregistered [ 8019.414899] LNet: 266530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8019.439391] LNetError: 266530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8019.487292] LNet: Removed LNI 192.168.201.111@tcp [ 8020.722748] Key type .llcrypt unregistered [ 8020.725901] Key type ._llcrypt unregistered [ 8043.004162] Key type ._llcrypt registered [ 8043.012237] Key type .llcrypt registered [ 8043.109491] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8044.905214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8045.088001] alg: No test for adler32 (adler32-zlib) [ 8046.341462] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 8046.674401] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8048.416185] Key type lgssc registered [ 8049.458841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8057.392968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8057.431381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.965913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8064.651256] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8069.089728] LustreError: 267754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.100078] LustreError: 267754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8074.222314] LustreError: 267755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8075.677453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8076.176245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8080.958415] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8090.439875] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8098.438274] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8105.364330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.788775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.846947] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8121.286820] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8129.340753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8141.794876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8141.806242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.830395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8144.865413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8144.874542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8144.903124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8147.576754] Lustre: server umount lustre-OST0000 complete [ 8153.058715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8153.067718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8153.086317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8153.091040] Lustre: Skipped 1 previous similar message [ 8155.105606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.107374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8160.225173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8160.250579] Lustre: Skipped 2 previous similar messages [ 8166.369303] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8166.827319] Lustre: server umount lustre-MDT0000 complete [ 8170.468231] LustreError: 267755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8170.484390] LustreError: 267755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8171.781207] LustreError: 267735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786439834 with bad export cookie 5684162715998340284 [ 8171.796215] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8171.797306] LustreError: 267735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8172.417953] Lustre: server umount lustre-MDT0001 complete [ 8188.106252] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8191.829457] Key type lgssc unregistered [ 8192.270105] LNet: 271462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8192.274498] LNetError: 271462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8193.318525] LNet: Removed LNI 192.168.201.111@tcp [ 8194.519961] Key type .llcrypt unregistered [ 8194.521930] Key type ._llcrypt unregistered [ 8224.690533] Key type ._llcrypt registered [ 8224.695693] Key type .llcrypt registered [ 8224.859667] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8226.264897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8226.350345] alg: No test for adler32 (adler32-zlib) [ 8227.634582] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 8227.976948] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8229.816254] Key type lgssc registered [ 8231.149032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8240.500042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8240.541360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8242.196288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.201593] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8252.392017] LustreError: 272880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8252.415268] LustreError: 272880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8257.406446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8257.507623] LustreError: 272881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8257.982461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8264.177897] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8272.569327] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8279.133191] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8286.517806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.879371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8287.921844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8293.693585] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8300.972594] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8329.186090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8329.190230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.229726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8329.240617] Lustre: Skipped 1 previous similar message [ 8334.314325] LustreError: 274701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8334.338706] LustreError: 274701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8334.421818] Lustre: server umount lustre-OST0000 complete [ 8339.431343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.440449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8339.451881] Lustre: Skipped 2 previous similar messages [ 8339.480988] Lustre: Skipped 1 previous similar message [ 8344.547388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8344.563851] Lustre: Skipped 1 previous similar message [ 8345.189955] Lustre: server umount lustre-MDT0000 complete [ 8349.320561] LustreError: 272863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786440011 with bad export cookie 11574445387797209392 [ 8349.324220] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8349.333859] LustreError: 272863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8349.777591] Lustre: server umount lustre-MDT0001 complete [ 8364.511424] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8368.974797] Key type lgssc unregistered [ 8369.373683] LNet: 277193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8369.386800] LNetError: 277193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8369.421136] LNet: Removed LNI 192.168.201.111@tcp [ 8370.633945] Key type .llcrypt unregistered [ 8370.636125] Key type ._llcrypt unregistered [ 8391.549609] Key type ._llcrypt registered [ 8391.552226] Key type .llcrypt registered [ 8391.677865] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8393.178842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8393.303238] alg: No test for adler32 (adler32-zlib) [ 8394.383946] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 8394.675066] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8396.368157] Key type lgssc registered [ 8397.666831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8406.298308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8406.318293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.848319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.862478] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8418.279855] LustreError: 278429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8418.304937] LustreError: 278429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8422.016814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.436899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.353612] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8435.853945] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8442.412353] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8450.322836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8450.956630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8452.987561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8458.691355] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8466.920359] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8482.795978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8482.798477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.816892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.831736] Lustre: Skipped 1 previous similar message [ 8482.852214] Lustre: Skipped 1 previous similar message [ 8485.846913] Lustre: server umount lustre-OST0000 complete [ 8493.029917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8493.056865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8493.065570] Lustre: Skipped 1 previous similar message [ 8496.733269] Lustre: server umount lustre-MDT0000 complete [ 8498.152472] LustreError: 278432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8498.167269] LustreError: 278432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8501.388653] LustreError: 278395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786440163 with bad export cookie 17915760400406907043 [ 8501.400139] LustreError: 278395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8501.403894] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8501.702038] Lustre: server umount lustre-MDT0001 complete [ 8515.301195] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8519.193673] Key type lgssc unregistered [ 8519.571324] LNet: 282133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8519.577744] LNetError: 282133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8519.609474] LNet: Removed LNI 192.168.201.111@tcp [ 8520.950715] Key type .llcrypt unregistered [ 8520.955658] Key type ._llcrypt unregistered [ 8532.191656] Key type ._llcrypt registered [ 8532.193299] Key type .llcrypt registered [ 8532.269109] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8550.841520] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 8567.084027] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8568.714552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8568.809745] alg: No test for adler32 (adler32-zlib) [ 8569.884906] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 8570.067770] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8571.736254] Key type lgssc registered [ 8573.414849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8610.129274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8610.174120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.403781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8611.441891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8611.526507] Lustre: lustre-MDT0000: new disk, initializing [ 8611.597605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.608846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8615.801343] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8627.282683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.469733] Lustre: 285688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8627.519763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8627.523248] Lustre: Skipped 1 previous similar message [ 8627.571645] Lustre: lustre-MDT0001: new disk, initializing [ 8627.641849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8627.676617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8627.686584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8632.018526] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8642.766436] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8649.318561] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8656.784590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8657.029703] Lustre: lustre-OST0000: new disk, initializing [ 8657.033373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8657.038480] Lustre: 287655:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8657.133167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8664.369545] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8665.707642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8665.712983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8665.807335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8673.525807] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8678.747921] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8679.035734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8684.174644] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8684.536727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8691.690979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8691.693210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.702226] LustreError: Skipped 1 previous similar message [ 8691.704525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8691.719987] Lustre: Skipped 1 previous similar message [ 8691.742262] Lustre: Skipped 1 previous similar message [ 8692.980627] Lustre: server umount lustre-OST0000 complete [ 8699.361768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8699.376158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.402939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8701.921481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8701.922260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.520810] Lustre: server umount lustre-MDT0000 complete [ 8707.603344] LustreError: 285678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786440369 with bad export cookie 10812386246821093043 [ 8707.614723] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8707.615132] LustreError: 285678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8707.891450] Lustre: server umount lustre-MDT0001 complete [ 8717.785751] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:26:18 (1786440378) [ 8724.554663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8725.157610] LustreError: 290069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8725.170162] LustreError: 290069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8725.211583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8728.556572] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8730.598341] LustreError: 290070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8735.732076] LustreError: 290069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8736.630494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8737.144096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8741.887565] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8749.032719] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8754.423397] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8761.247927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8761.690558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8768.060378] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8775.656813] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8785.378880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8785.392235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.394721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8785.402761] Lustre: Skipped 1 previous similar message [ 8785.412289] Lustre: Skipped 2 previous similar messages [ 8790.507994] LustreError: 291890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.656900] Lustre: server umount lustre-OST0000 complete [ 8795.617771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8795.621832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8795.636607] Lustre: Skipped 1 previous similar message [ 8795.650460] Lustre: Skipped 1 previous similar message [ 8803.041723] Lustre: server umount lustre-MDT0000 complete [ 8805.860544] LustreError: 290797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8805.886765] LustreError: 290797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8807.390326] LustreError: 290049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786440469 with bad export cookie 10812386246821093925 [ 8807.409868] LustreError: 290049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8807.411060] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8807.965747] Lustre: server umount lustre-MDT0001 complete [ 8818.709233] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8820.781581] Key type lgssc unregistered [ 8821.162738] LNet: 293607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8821.183192] LNetError: 293607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8821.204187] LNet: Removed LNI 192.168.201.111@tcp [ 8821.948917] Key type .llcrypt unregistered [ 8821.953675] Key type ._llcrypt unregistered [ 8854.216945] Key type ._llcrypt registered [ 8854.218466] Key type .llcrypt registered [ 8854.382469] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 8870.244797] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8871.368758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8871.442152] alg: No test for adler32 (adler32-zlib) [ 8872.668182] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 8872.933212] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8874.672831] Key type lgssc registered [ 8875.750118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8921.358407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8921.391839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.628091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8922.660197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8922.747277] Lustre: lustre-MDT0000: new disk, initializing [ 8922.812507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8922.835414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8927.704504] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8938.835409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8938.969053] Lustre: 297435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8939.029136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8939.033855] Lustre: Skipped 1 previous similar message [ 8939.112909] Lustre: lustre-MDT0001: new disk, initializing [ 8939.232848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8939.271430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8939.281283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8944.696381] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8955.087498] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8961.247531] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8968.482990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8968.774978] Lustre: lustre-OST2710: new disk, initializing [ 8968.779885] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8968.786389] Lustre: 299403:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8968.924547] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8974.802189] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8978.010293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8978.029212] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8978.268759] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8984.672432] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8991.748982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8991.994662] Lustre: lustre-OST03e8: new disk, initializing [ 8992.003550] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8992.007991] Lustre: 300467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8992.100630] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8999.127785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8999.140363] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8999.297544] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8999.332511] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9005.804518] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9012.786731] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9012.969558] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9017.790836] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9018.007832] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9022.660031] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9022.976258] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9027.952856] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9028.182495] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9035.239168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9035.251127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9035.256885] Lustre: Skipped 1 previous similar message [ 9035.274910] Lustre: Skipped 3 previous similar messages [ 9035.380676] Lustre: server umount lustre-MDT0000 complete [ 9038.677261] LustreError: 297426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786440700 with bad export cookie 12767591704569033941 [ 9038.679896] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9038.691177] LustreError: 297426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9039.152741] Lustre: server umount lustre-MDT0001 complete [ 9052.063685] Lustre: server umount lustre-OST2710 complete [ 9065.357201] Lustre: server umount lustre-OST03e8 complete [ 9083.086731] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 9094.231621] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9132.848873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.063674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9133.083708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9133.145502] Lustre: lustre-MDT0000: new disk, initializing [ 9133.207618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9133.222174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9137.423649] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9147.346187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9147.466641] Lustre: 305719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9147.551547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9147.562113] Lustre: Skipped 1 previous similar message [ 9147.636807] Lustre: lustre-MDT0001: new disk, initializing [ 9147.766327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9147.817415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9147.843108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9152.324936] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9161.046978] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9166.509393] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9172.962110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9173.264645] Lustre: lustre-OST0000: new disk, initializing [ 9173.276402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9173.298273] Lustre: 307686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9173.395571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.561603] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9182.313477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9182.328127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9182.458754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9188.413400] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9192.595137] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9192.799932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9197.476264] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9197.725475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9203.182727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9203.193118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9203.215464] Lustre: Skipped 2 previous similar messages [ 9203.229100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9203.238784] Lustre: Skipped 1 previous similar message [ 9205.553481] Lustre: server umount lustre-OST0000 complete [ 9209.314776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9209.317665] LustreError: Skipped 1 previous similar message [ 9209.330830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9209.346838] Lustre: Skipped 1 previous similar message [ 9209.357151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9213.413763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9213.414439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9214.984198] Lustre: server umount lustre-MDT0000 complete [ 9218.267253] LustreError: 305711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786440880 with bad export cookie 12767591704569035621 [ 9218.277211] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9218.278529] LustreError: 305711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9218.640907] Lustre: server umount lustre-MDT0001 complete [ 9226.656448] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:34:47 (1786440887) [ 9228.069107] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9229.402828] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:34:50 (1786440890) [ 9233.091457] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9258.548727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9258.778294] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9258.795603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9258.930427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9264.083106] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9275.671214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.864443] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9275.934788] Lustre: 311545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9282.226995] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9304.303215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9304.535796] Lustre: Denying initial registration attempt from nid 192.168.201.111@tcp, specified as failover [ 9304.554662] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9304.564855] LustreError: 313000:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9304.574720] LustreError: 313000:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9304.580680] LustreError: 313000:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9304.662417] Lustre: server umount lustre-OST0000 complete [ 9304.664348] LustreError: 313000:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9312.236618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9312.247380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.275522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9312.283819] Lustre: Skipped 1 previous similar message [ 9321.446914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9321.450183] Lustre: Skipped 2 previous similar messages [ 9326.048286] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9326.213119] Lustre: server umount lustre-MDT0000 complete [ 9330.520040] LustreError: 313370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786440992 with bad export cookie 12767591704569036496 [ 9330.531404] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9330.534953] LustreError: 313370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9330.974304] Lustre: server umount lustre-MDT0001 complete [ 9343.185328] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 9346.365905] Key type lgssc unregistered [ 9346.674663] LNet: 314126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9346.683413] LNetError: 314126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9346.703898] LNet: Removed LNI 192.168.201.111@tcp [ 9347.607042] Key type .llcrypt unregistered [ 9347.613616] Key type ._llcrypt unregistered [ 9359.885490] Key type ._llcrypt registered [ 9359.886939] Key type .llcrypt registered [ 9360.005849] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9361.174899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9361.211987] alg: No test for adler32 (adler32-zlib) [ 9362.389136] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 9362.724551] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 9364.466549] Key type lgssc registered [ 9366.165966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9416.327540] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9425.158535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9425.166823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9426.360565] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9426.398791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9426.636826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9430.847263] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9438.585143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.732548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9438.763535] Lustre: 316891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9439.074391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9444.248246] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9452.097211] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9458.122596] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9478.524559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9478.767426] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9478.939305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9485.386850] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9493.397376] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9499.621082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9499.628466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9499.656301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9499.660308] Lustre: Skipped 1 previous similar message [ 9504.744134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9504.747831] Lustre: Skipped 1 previous similar message [ 9509.863449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9509.880341] Lustre: Skipped 1 previous similar message [ 9510.368298] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9510.746323] Lustre: server umount lustre-OST0000 complete [ 9516.008481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9516.019652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9516.029154] Lustre: Skipped 1 previous similar message [ 9516.034657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9520.100076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9520.102089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9520.127259] Lustre: Skipped 1 previous similar message [ 9521.786790] Lustre: server umount lustre-MDT0000 complete [ 9525.832641] LustreError: 316885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786441188 with bad export cookie 6587760774317909471 [ 9525.848253] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9526.183773] Lustre: server umount lustre-MDT0001 complete [ 9537.995916] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 9540.713669] Key type lgssc unregistered [ 9540.972996] LNet: 320470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9540.979069] LNetError: 320470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9540.990717] LNet: Removed LNI 192.168.201.111@tcp [ 9541.767996] Key type .llcrypt unregistered [ 9541.769777] Key type ._llcrypt unregistered [ 9566.638731] Key type ._llcrypt registered [ 9566.646424] Key type .llcrypt registered [ 9566.784176] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9568.453777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9568.506145] alg: No test for adler32 (adler32-zlib) [ 9569.562965] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 9569.871821] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 9571.528262] Key type lgssc registered [ 9572.622311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9580.042779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9580.073773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9581.649727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9586.622781] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9591.783218] LustreError: 321888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9591.808760] LustreError: 321888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9594.293298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9594.660498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9599.276811] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9606.439418] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9612.293513] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9621.293897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9621.790938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9628.145850] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9636.482110] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9649.159813] Lustre: server umount lustre-MDT0000 complete [ 9649.642610] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9649.667041] LustreError: 321887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9649.690529] LustreError: 321887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9652.933382] LustreError: 321867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786441315 with bad export cookie 1880385117703602011 [ 9652.944748] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9652.948292] LustreError: 321867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9653.845703] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.11@tcp (stopping) [ 9653.859623] LustreError: 323541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9654.761390] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9654.776339] Lustre: Skipped 2 previous similar messages [ 9654.786152] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9657.761627] LustreError: 321883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.764791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9657.792988] LustreError: 321883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9659.174848] Lustre: server umount lustre-MDT0001 complete [ 9665.729200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9675.223459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.539306] LustreError: 323722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.546747] LustreError: 323722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9678.601759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9678.652700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9684.314884] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9687.852916] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9687.861359] LustreError: 325489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.869672] LustreError: 325489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9693.074202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9693.170749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9693.179303] Lustre: Skipped 1 previous similar message [ 9693.334891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9698.069407] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9698.805904] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9705.365193] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9706.897664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9713.223788] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9714.539147] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9723.900135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9723.908809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9723.938479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9726.835605] Lustre: server umount lustre-OST0000 complete [ 9734.119930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9734.132091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9734.160985] Lustre: Skipped 1 previous similar message [ 9734.185751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9734.191879] Lustre: Skipped 2 previous similar messages [ 9737.486131] Lustre: server umount lustre-MDT0000 complete [ 9739.747565] LustreError: 325491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.806195] LustreError: 325491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9743.173160] LustreError: 321867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786441405 with bad export cookie 1880385117703609179 [ 9743.176860] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9743.193119] LustreError: 321867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9743.779460] Lustre: server umount lustre-MDT0001 complete [ 9755.478797] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 9759.550574] Key type lgssc unregistered [ 9759.869090] LNet: 328166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9759.874696] LNetError: 328166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9759.885606] LNet: Removed LNI 192.168.201.111@tcp [ 9760.840909] Key type .llcrypt unregistered [ 9760.843034] Key type ._llcrypt unregistered [ 9785.140384] Key type ._llcrypt registered [ 9785.142445] Key type .llcrypt registered [ 9785.232731] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9786.531586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9786.555120] alg: No test for adler32 (adler32-zlib) [ 9787.722167] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [ 9787.992380] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 9789.760168] Key type lgssc registered [ 9790.888769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9798.627924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9798.641098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9800.127172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9804.902566] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9810.403470] LustreError: 329587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.416361] LustreError: 329587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9813.036414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9813.439425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9817.453674] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9824.426184] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9829.980024] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9833.958668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9833.973496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9834.000225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9836.001189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9836.004904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9838.113282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9838.127981] Lustre: Skipped 2 previous similar messages [ 9843.169744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9848.291792] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9848.299342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9848.310117] Lustre: Skipped 2 previous similar messages [ 9849.075505] Lustre: server umount lustre-MDT0000 complete [ 9853.085439] LustreError: 329567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786441515 with bad export cookie 7720155828310460037 [ 9853.103355] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9853.107642] LustreError: 329567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9853.465602] Lustre: server umount lustre-MDT0001 complete [ 9866.244554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.664720] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9866.692842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9866.853733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9869.833881] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9877.803559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9877.898211] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9877.948482] Lustre: 332173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9878.150781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9882.145575] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9891.171550] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9896.431373] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9906.090210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9906.369350] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9906.377279] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9906.382303] LustreError: 334056:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9906.391616] LustreError: 334056:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9906.396977] LustreError: 334056:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9906.460324] Lustre: server umount lustre-OST0000 complete [ 9906.462563] LustreError: 334056:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9917.048966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9917.306702] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9917.582187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9924.189926] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9925.715275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9931.256791] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9938.732048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9938.860115] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9938.938864] Lustre: lustre-OST0001: new disk, initializing [ 9938.949584] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9938.955948] Lustre: 335598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9939.098789] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9944.939844] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9946.803806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9946.812866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9946.968838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9952.250984] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9957.347106] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9957.355631] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9957.370814] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9957.378297] Lustre: Skipped 1 previous similar message [ 9961.195914] Lustre: server umount lustre-OST0001 complete [ 9962.484563] LustreError: 334522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9962.497951] LustreError: 334522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9966.562442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9966.565205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9966.567324] LustreError: Skipped 2 previous similar messages [ 9966.581576] Lustre: Skipped 1 previous similar message [ 9967.592574] LustreError: 334523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9967.604216] LustreError: 334523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9971.116246] Lustre: server umount lustre-OST0000 complete [ 9975.266190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9975.270393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.299945] Lustre: Skipped 1 previous similar message [ 9975.305441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9975.316275] Lustre: Skipped 4 previous similar messages [ 9980.888513] Lustre: server umount lustre-MDT0000 complete [ 9982.946452] LustreError: 332179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9982.959678] LustreError: 332179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9984.468437] LustreError: 332167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786441646 with bad export cookie 7720155828310460590 [ 9984.471399] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9984.475615] LustreError: 332167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9984.714493] Lustre: server umount lustre-MDT0001 complete [ 9994.965911] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 9997.401693] Key type lgssc unregistered [ 9997.694715] LNet: 337557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9997.704059] LNetError: 337557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9997.718755] LNet: Removed LNI 192.168.201.111@tcp [ 9998.512530] Key type .llcrypt unregistered [ 9998.515221] Key type ._llcrypt unregistered [10073.037322] Key type ._llcrypt registered [10073.039447] Key type .llcrypt registered [10073.132342] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [10087.058865] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10088.283913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10088.331613] alg: No test for adler32 (adler32-zlib) [10089.387225] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [10089.623174] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [10091.385822] Key type lgssc registered [10092.585339] Lustre: Echo OBD driver; http://www.lustre.org/ [10123.094648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10123.123934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10124.403382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10124.433427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10124.501234] Lustre: lustre-MDT0000: new disk, initializing [10124.569579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10124.584623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10129.166908] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10139.565428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10139.688285] Lustre: 342492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10139.746285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10139.750197] Lustre: Skipped 1 previous similar message [10139.822992] Lustre: lustre-MDT0001: new disk, initializing [10139.955686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10140.013403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10140.030163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10144.969372] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10154.463334] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10159.562902] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10166.249893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10166.547486] Lustre: lustre-OST0000: new disk, initializing [10166.552784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10166.559519] Lustre: 344462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10166.677643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10172.707803] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10174.012825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10174.024786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10174.188959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10181.109401] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10185.410789] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10185.651271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10190.636951] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10190.814851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10194.917328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10194.946265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10194.953906] Lustre: Skipped 1 previous similar message [10194.978402] Lustre: Skipped 1 previous similar message [10198.884175] Lustre: server umount lustre-OST0000 complete [10205.154439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10205.168117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10205.172203] Lustre: Skipped 1 previous similar message [10209.061490] Lustre: server umount lustre-MDT0000 complete [10212.569173] LustreError: 344461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786441874 with bad export cookie 7663836862973251842 [10212.573794] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10212.582937] LustreError: 344461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10213.016261] Lustre: server umount lustre-MDT0001 complete [10220.752601] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:51:21 (1786441881) [10227.757541] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:51:28 (1786441888) [10235.388425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.799914] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.822479] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10235.881524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10240.958952] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10241.002634] LustreError: 347218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10246.119931] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.286437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10250.545354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10255.108583] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10264.697105] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10270.450758] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10278.355938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10278.995645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10285.742572] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10294.504357] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10302.386788] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10303.773376] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10305.984143] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10308.374355] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10310.155164] Lustre: Failing over lustre-MDT0000 [10310.449589] Lustre: server umount lustre-MDT0000 complete [10310.627052] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10310.642817] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10310.645104] Lustre: Skipped 2 previous similar messages [10310.669208] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10315.748791] LustreError: 347214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10315.784174] LustreError: 347214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10325.999534] LustreError: 347212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10326.022854] LustreError: 347212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10330.549214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.696147] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10331.036446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10331.079935] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10332.238566] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10335.540921] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10336.239336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10336.270235] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10336.297643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10344.241771] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10345.773402] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10349.726415] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10352.454903] Lustre: Failing over lustre-MDT0000 [10352.718960] LustreError: 349062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10352.729745] LustreError: 349062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10354.801684] Lustre: server umount lustre-MDT0000 complete [10356.711298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10356.734357] Lustre: Skipped 2 previous similar messages [10372.066678] Lustre: 340717:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786442018/real 1786442018] req@0000000031207f39 x1873219945586816/t0(0) o400->MGC192.168.201.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786442034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10372.097270] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10375.578951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.310751] LustreError: 340714:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000091dd435e x1873219945592064/t0(0) o250->MGC192.168.201.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10382.787199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10382.894179] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10383.436329] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10387.957516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10387.962507] Lustre: Skipped 2 previous similar messages [10388.027276] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10388.062124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10388.227053] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10398.213417] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10399.801676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10408.417432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.433797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10408.441462] Lustre: Skipped 3 previous similar messages [10408.467439] Lustre: Skipped 2 previous similar messages [10412.826606] Lustre: server umount lustre-MDT0000 complete [10413.539757] LustreError: 347218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.559762] LustreError: 347218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [10416.646942] LustreError: 347198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786442078 with bad export cookie 7663836862973254558 [10416.652548] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10416.662204] LustreError: 347198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10417.040870] Lustre: server umount lustre-MDT0001 complete [10431.409885] Lustre: server umount lustre-OST0000 complete [10446.810972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10447.193757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10451.260910] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10459.546841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.694247] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10472.906751] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10479.370522] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10487.766724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10492.419830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10495.062307] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10503.102978] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10514.922925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [10514.931461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10514.948749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10515.944645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10521.059211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10521.065097] Lustre: Skipped 3 previous similar messages [10528.224143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10528.424285] Lustre: server umount lustre-OST0000 complete [10536.419513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10536.443097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10536.447386] Lustre: Skipped 2 previous similar messages [10536.470309] Lustre: Skipped 3 previous similar messages [10538.823422] Lustre: server umount lustre-MDT0000 complete [10541.537574] LustreError: 353560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10541.551253] LustreError: 353560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10543.290252] LustreError: 353545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786442205 with bad export cookie 7663836862973255300 [10543.299634] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10543.316357] LustreError: 353545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10543.612960] Lustre: server umount lustre-MDT0001 complete [10556.477289] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [10560.041991] Key type lgssc unregistered [10560.397930] LNet: 357102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10560.403457] LNetError: 357102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10560.420880] LNet: Removed LNI 192.168.201.111@tcp [10561.319365] Key type .llcrypt unregistered [10561.320967] Key type ._llcrypt unregistered [10595.028764] Key type ._llcrypt registered [10595.031598] Key type .llcrypt registered [10595.131305] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10596.240939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10596.329919] alg: No test for adler32 (adler32-zlib) [10597.434954] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [10597.743541] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [10599.521648] Key type lgssc registered [10600.604650] Lustre: Echo OBD driver; http://www.lustre.org/ [10610.201580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10610.240851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10612.284248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10617.799614] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10622.439864] LustreError: 358619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10622.457643] LustreError: 358619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10627.556259] LustreError: 358620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10628.000632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10628.326748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10633.694497] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10642.723116] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10650.129918] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10657.530107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10657.905751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10658.938907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10663.848677] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10673.092754] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10684.905592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10684.913358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10684.934608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10688.996744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10689.013667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10689.045483] Lustre: Skipped 1 previous similar message [10694.115440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10694.126264] Lustre: Skipped 4 previous similar messages [10699.232113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10699.240268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10699.249352] Lustre: Skipped 2 previous similar messages [10699.492519] Lustre: server umount lustre-MDT0000 complete [10704.124567] LustreError: 358601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786442366 with bad export cookie 3498830608399353934 [10704.141931] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10704.142829] LustreError: 358601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10704.365701] LustreError: 360462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10704.378464] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10704.408277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10704.666427] Lustre: server umount lustre-MDT0001 complete [10711.666311] Lustre: server umount lustre-OST0000 complete [10725.085542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.318753] LustreError: 362336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.355974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10730.467963] LustreError: 362337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10730.751477] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10739.716620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10740.310617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10745.914449] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10754.606053] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10761.682048] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10770.499408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10771.070268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10778.320743] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10786.754444] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10795.653548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10811.368262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.373362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10811.391179] Lustre: Skipped 1 previous similar message [10811.413037] Lustre: Skipped 2 previous similar messages [10816.400245] Lustre: server umount lustre-MDT0000 complete [10816.487177] LustreError: 363065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.516759] LustreError: 363065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10820.557465] LustreError: 364181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786442482 with bad export cookie 3498830608399355054 [10820.567402] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.575090] LustreError: 364181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10820.970996] Lustre: server umount lustre-MDT0001 complete [10836.015154] Lustre: server umount lustre-OST0000 complete [10852.878850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10853.446695] LustreError: 366281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10853.560542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10858.786977] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10867.304575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10872.821346] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10881.689378] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10888.237464] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10896.344419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10896.567968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10896.571089] Lustre: Skipped 1 previous similar message [10898.632751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10903.084260] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10909.929473] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10917.349556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10917.351773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10917.375286] Lustre: Skipped 2 previous similar messages [10917.389414] Lustre: Skipped 1 previous similar message [10917.929676] Lustre: server umount lustre-OST0000 complete [10928.273419] Lustre: server umount lustre-MDT0000 complete [10932.532105] LustreError: 366263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786442594 with bad export cookie 3498830608399356076 [10932.535881] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10932.544755] LustreError: 366263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10932.839293] Lustre: server umount lustre-MDT0001 complete [10947.957677] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [10952.000894] Key type lgssc unregistered [10952.565720] LNet: 369818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10952.580714] LNetError: 369818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10952.598057] LNet: Removed LNI 192.168.201.111@tcp [10953.561151] Key type .llcrypt unregistered [10953.569812] Key type ._llcrypt unregistered [10984.158940] Key type ._llcrypt registered [10984.161714] Key type .llcrypt registered [10984.291539] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10985.839773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10985.862973] alg: No test for adler32 (adler32-zlib) [10987.108334] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [10987.458932] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [10989.164909] Key type lgssc registered [10990.235995] Lustre: Echo OBD driver; http://www.lustre.org/ [10999.024138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10999.053063] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10999.062257] LustreError: 371258:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10999.090242] LustreError: 371258:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10999.097119] LustreError: 371258:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10999.103715] LustreError: 371258:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10999.113488] LustreError: 371258:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11007.764310] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11007.775632] LustreError: 371655:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11007.797712] LustreError: 371655:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11007.804365] LustreError: 371655:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11007.811875] LustreError: 371655:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11007.818292] LustreError: 371655:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11032.651840] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11036.084699] Key type lgssc unregistered [11036.417741] LNet: 372664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11036.423490] LNetError: 372664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11036.455375] LNet: Removed LNI 192.168.201.111@tcp [11037.398431] Key type .llcrypt unregistered [11037.412566] Key type ._llcrypt unregistered [11059.348534] Key type ._llcrypt registered [11059.352253] Key type .llcrypt registered [11059.528928] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [11080.087142] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11081.980623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11082.150342] alg: No test for adler32 (adler32-zlib) [11083.089649] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [11083.252899] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11085.050153] Key type lgssc registered [11087.395103] Lustre: Echo OBD driver; http://www.lustre.org/ [11124.808770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11124.879536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11126.262514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11126.292856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11126.398583] Lustre: lustre-MDT0000: new disk, initializing [11126.548649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11126.592632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11132.685429] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11145.731265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11145.873958] Lustre: 375824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11145.930097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11145.933652] Lustre: Skipped 1 previous similar message [11146.015756] Lustre: lustre-MDT0001: new disk, initializing [11146.194742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11146.251621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11146.270543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11152.071806] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11163.802270] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11170.649219] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11178.101408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11178.453961] Lustre: lustre-OST0000: new disk, initializing [11178.457746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11178.467381] Lustre: 377792:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11178.557971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11183.812661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11183.839039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11184.059743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11186.117831] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11195.523746] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11202.468473] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11202.699205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11209.621732] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11210.071801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11214.819392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11214.824486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11214.833103] Lustre: Skipped 1 previous similar message [11214.839450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11214.843650] Lustre: Skipped 1 previous similar message [11218.707342] Lustre: server umount lustre-OST0000 complete [11223.025563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11223.036529] LustreError: Skipped 1 previous similar message [11223.050230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11223.065955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11225.056988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11225.060795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11228.979341] Lustre: server umount lustre-MDT0000 complete [11233.427429] LustreError: 376618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786442895 with bad export cookie 162099737248412692 [11233.434284] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11233.443604] LustreError: 376618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11233.981251] Lustre: server umount lustre-MDT0001 complete [11245.175209] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11247.451912] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:08:27 (1786442907) [11254.851390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11255.310698] LustreError: 380254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11255.326767] LustreError: 380254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11255.389329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11260.387304] LustreError: 380255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11261.255353] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11265.510118] LustreError: 380254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.632123] LustreError: 380255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11273.835512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11274.507562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11280.144382] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11289.292703] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11295.852596] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11302.603349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11302.944283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11310.063980] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11317.713952] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11327.024614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11327.319957] Lustre: lustre-OST0001: new disk, initializing [11327.328570] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11327.341366] Lustre: 383093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11327.439701] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11334.485513] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11337.402584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11337.415965] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11337.571366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11346.105706] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11353.064038] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11353.072359] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11353.101407] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11353.107257] Lustre: Skipped 1 previous similar message [11353.121886] Lustre: Skipped 2 previous similar messages [11356.360330] Lustre: server umount lustre-OST0001 complete [11358.192025] LustreError: 382075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.217454] LustreError: 382075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11368.423505] LustreError: 382078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11368.438764] LustreError: 382078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11375.585633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11375.590859] LustreError: Skipped 1 previous similar message [11375.594175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.614779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11375.617420] Lustre: Skipped 1 previous similar message [11380.653798] Lustre: server umount lustre-OST0000 complete [11387.361405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11387.368890] LustreError: Skipped 1 previous similar message [11387.371880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11387.381743] Lustre: Skipped 1 previous similar message [11387.388391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11387.393339] Lustre: Skipped 3 previous similar messages [11391.887217] Lustre: server umount lustre-MDT0000 complete [11394.025170] LustreError: 383655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11394.052961] LustreError: 383655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11395.575393] LustreError: 382100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786443057 with bad export cookie 162099737248413574 [11395.576502] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11395.584848] LustreError: 382100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11402.059702] Lustre: server umount lustre-MDT0001 complete [11416.343721] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11419.428545] Key type lgssc unregistered [11419.877634] LNet: 385085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11419.882093] LNetError: 385085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11419.892144] LNet: Removed LNI 192.168.201.111@tcp [11420.757828] Key type .llcrypt unregistered [11420.759867] Key type ._llcrypt unregistered [11474.637292] Key type ._llcrypt registered [11474.642167] Key type .llcrypt registered [11474.732327] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11476.300789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11476.332752] alg: No test for adler32 (adler32-zlib) [11477.569590] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [11477.904300] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11479.696504] Key type lgssc registered [11480.940337] Lustre: Echo OBD driver; http://www.lustre.org/ [11488.849732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11488.864123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11490.193732] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11490.211361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11490.340433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11496.248826] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11505.403054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11505.523800] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11505.552610] Lustre: 387887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11505.875326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11510.408284] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11518.032656] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11523.959985] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11526.115786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11526.117721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11526.129642] Lustre: Skipped 1 previous similar message [11526.139812] Lustre: Skipped 1 previous similar message [11531.234154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11531.245678] Lustre: Skipped 1 previous similar message [11532.419595] Lustre: server umount lustre-MDT0000 complete [11535.717813] LustreError: 387880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786443197 with bad export cookie 2890966546509911925 [11535.729636] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11535.735910] LustreError: 387880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11536.413946] Lustre: server umount lustre-MDT0001 complete [11544.575226] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:13:25 (1786443205) [11549.706903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11550.140590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11553.996380] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11555.300950] LustreError: 390398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11555.313790] LustreError: 390398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11560.430194] LustreError: 390399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11561.304394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11561.581291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11566.342767] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11573.655332] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11579.483981] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11585.706607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11585.919611] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11586.109795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11592.677546] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11600.704365] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11608.807946] Lustre: Permanently deactivating lustre-OST0000 [11608.819715] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11616.358149] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11618.218965] LustreError: 393383:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11618.236777] LustreError: 393383:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11618.245900] LustreError: 393383:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11620.920161] LustreError: 393432:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11620.924740] LustreError: 393432:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11628.737276] Lustre: server umount lustre-OST0000 complete [11631.419167] LustreError: 393681:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11631.433211] LustreError: 393681:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11636.705364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11636.711311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11636.732901] Lustre: Skipped 1 previous similar message [11641.827596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11641.834092] Lustre: Skipped 2 previous similar messages [11642.932183] Lustre: server umount lustre-MDT0000 complete [11646.114809] LustreError: 390378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786443308 with bad export cookie 2890966546509912506 [11646.123946] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11656.946341] Lustre: server umount lustre-MDT0001 complete [11666.852906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.239333] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11673.852235] LustreError: 394841:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11680.340190] Lustre: 395181:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.111@tcp [11682.671734] Lustre: server umount MGS complete [11693.285061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11693.759689] LustreError: 395905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.779817] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11693.784970] Lustre: Skipped 1 previous similar message [11693.826154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11698.026776] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11699.178143] LustreError: 395906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11704.303571] LustreError: 395905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.431783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11706.673480] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11711.574707] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11718.783157] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11723.639427] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11725.411161] Lustre: Permanently reactivating lustre-OST0000 [11725.422714] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11725.428120] Lustre: Skipped 2 previous similar messages [11752.416556] LustreError: 397665:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11758.627663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11758.910859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11758.915255] Lustre: Skipped 1 previous similar message [11764.151919] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11770.880576] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11777.510251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11777.513096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11777.514540] LustreError: Skipped 1 previous similar message [11777.520092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11777.524049] Lustre: Skipped 1 previous similar message [11777.553372] Lustre: Skipped 1 previous similar message [11781.027670] Lustre: server umount lustre-OST0000 complete [11787.748680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11787.753100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11787.765022] Lustre: Skipped 1 previous similar message [11787.791708] Lustre: Skipped 1 previous similar message [11790.682379] Lustre: server umount lustre-MDT0000 complete [11792.865676] LustreError: 395905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11792.879584] LustreError: 395905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11793.804943] LustreError: 395885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786443456 with bad export cookie 2890966546509913556 [11793.805982] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11793.811158] LustreError: 395885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11794.362311] Lustre: server umount lustre-MDT0001 complete [11805.694556] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11809.034831] Key type lgssc unregistered [11809.414222] LNet: 400318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11809.430697] LNetError: 400318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11809.449865] LNet: Removed LNI 192.168.201.111@tcp [11810.517642] Key type .llcrypt unregistered [11810.519412] Key type ._llcrypt unregistered [11827.738575] Key type ._llcrypt registered [11827.739922] Key type .llcrypt registered [11827.811606] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [11843.343309] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11844.357578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11844.383424] alg: No test for adler32 (adler32-zlib) [11845.534460] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [11845.780370] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11847.472196] Key type lgssc registered [11848.625900] Lustre: Echo OBD driver; http://www.lustre.org/ [11878.330626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11878.351220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11879.719925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11879.767741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11879.892077] Lustre: lustre-MDT0000: new disk, initializing [11879.998631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11880.016542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11884.207884] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11893.859617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.990026] Lustre: 403477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11894.044584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11894.053407] Lustre: Skipped 1 previous similar message [11894.166154] Lustre: lustre-MDT0001: new disk, initializing [11894.263706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11894.325216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11894.354272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11898.785560] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11907.427940] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11912.244586] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11918.528874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11918.753378] Lustre: lustre-OST0000: new disk, initializing [11918.756741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11918.762389] Lustre: 405440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11918.851538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11924.826984] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11927.651973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11927.664285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11927.844363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11934.151969] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11938.997288] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11939.159853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11943.035479] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11943.218886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11948.514917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11948.516669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11948.525611] LustreError: Skipped 1 previous similar message [11948.535184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11948.535904] Lustre: Skipped 1 previous similar message [11948.548171] Lustre: Skipped 1 previous similar message [11953.660528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11953.667474] Lustre: Skipped 1 previous similar message [11958.760307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11958.770863] Lustre: Skipped 1 previous similar message [11959.264270] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11959.454579] Lustre: server umount lustre-OST0000 complete [11963.876303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.887776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11963.902737] Lustre: Skipped 1 previous similar message [11968.996185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11969.006823] Lustre: Skipped 1 previous similar message [11969.242787] Lustre: server umount lustre-MDT0000 complete [11971.745453] LustreError: 403467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786443633 with bad export cookie 95533283694746513 [11971.746697] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11971.755983] LustreError: 403467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11972.065636] Lustre: server umount lustre-MDT0001 complete [11978.566784] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11980.213773] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:20:40 (1786443640) [11985.303337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11985.663534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11989.551890] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11991.015604] LustreError: 407906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.031449] LustreError: 407906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11995.107945] LustreError: 407905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11995.915915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11996.153520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11999.119631] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12004.164580] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12008.372170] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12014.264252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12014.506270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12019.318347] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12025.083863] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12026.762207] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12033.367500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12040.165693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12040.178602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12040.191822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12040.195664] Lustre: Skipped 2 previous similar messages [12040.210184] Lustre: Skipped 1 previous similar message [12043.485963] Lustre: server umount lustre-OST0000 complete [12047.329986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12047.336418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12052.385274] Lustre: server umount lustre-MDT0000 complete [12055.298639] LustreError: 407886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786443717 with bad export cookie 95533283694747395 [12055.299562] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12055.304333] LustreError: 407886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12055.517782] Lustre: server umount lustre-MDT0001 complete [12063.234325] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12065.516290] Key type lgssc unregistered [12065.757933] LNet: 411631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12065.762458] LNetError: 411631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12065.774222] LNet: Removed LNI 192.168.201.111@tcp [12066.451128] Key type .llcrypt unregistered [12066.452753] Key type ._llcrypt unregistered [12083.698392] Key type ._llcrypt registered [12083.702116] Key type .llcrypt registered [12083.775564] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12083.901409] Key type .llcrypt unregistered [12083.902495] Key type ._llcrypt unregistered [12098.074076] Key type ._llcrypt registered [12098.075829] Key type .llcrypt registered [12098.151728] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12098.790274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12098.797814] alg: No test for adler32 (adler32-zlib) [12099.932939] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [12100.161557] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [12101.832176] Key type lgssc registered [12102.811073] Lustre: Echo OBD driver; http://www.lustre.org/ [12109.396800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12109.420725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12110.700705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12114.590706] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12120.040987] LustreError: 413827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12120.056993] LustreError: 413827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12120.852378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12121.177651] LustreError: 413822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12121.187783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12121.190776] LustreError: 413822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12125.546824] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12126.177820] LustreError: 413827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12126.201572] LustreError: 413827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12130.888734] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12131.299831] LustreError: 413826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12131.313356] LustreError: 413826:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12135.401946] LustreError: 413827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.434947] LustreError: 413827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12135.455614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12136.596608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12136.867413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12140.477155] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12147.175029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12147.206549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12147.213273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12147.215702] Lustre: Skipped 1 previous similar message [12152.297053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12152.303360] Lustre: Skipped 1 previous similar message [12152.592369] Lustre: server umount lustre-OST0000 complete [12157.410095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12157.414585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.423942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12157.426869] Lustre: Skipped 2 previous similar messages [12162.492624] Lustre: server umount lustre-MDT0000 complete [12162.534680] LustreError: 415564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12162.543710] LustreError: 415564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12165.373285] LustreError: 413807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786443827 with bad export cookie 8348534242907819245 [12165.376057] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12165.380869] LustreError: 413807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12165.669261] Lustre: server umount lustre-MDT0001 complete [12173.909597] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12176.310273] Key type lgssc unregistered [12176.546253] LNet: 416977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12176.556515] LNetError: 416977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12177.575468] LNet: Removed LNI 192.168.201.111@tcp [12178.356726] Key type .llcrypt unregistered [12178.365248] Key type ._llcrypt unregistered [12197.090927] Key type ._llcrypt registered [12197.092425] Key type .llcrypt registered [12197.178726] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12198.189953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12198.232382] alg: No test for adler32 (adler32-zlib) [12199.257279] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [12199.372587] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [12201.000169] Key type lgssc registered [12201.979244] Lustre: Echo OBD driver; http://www.lustre.org/ [12207.581909] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12207.612222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12223.200418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12228.320478] LustreError: 418346:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.111@tcp: failed processing log, type 4: rc = -110 [12255.008207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12260.563860] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12268.270205] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12274.363122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12274.752087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12274.829965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12279.620927] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12279.787409] LustreError: 418372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12279.810301] LustreError: 418372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12284.906152] LustreError: 418371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12287.206237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12287.588767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12291.520191] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12297.522397] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12303.378426] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12310.498910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12310.503114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12310.503507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12310.509534] Lustre: Skipped 1 previous similar message [12315.412149] Lustre: server umount lustre-OST0000 complete [12320.741917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12320.760183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12320.766346] Lustre: Skipped 2 previous similar messages [12324.711164] Lustre: server umount lustre-MDT0000 complete [12325.857638] LustreError: 419386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12325.877214] LustreError: 419386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12327.914358] LustreError: 418352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786443990 with bad export cookie 9998932186052513491 [12327.916177] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12327.923898] LustreError: 418352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12328.169932] Lustre: server umount lustre-MDT0001 complete [12338.588197] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12341.143091] Key type lgssc unregistered [12341.380899] LNet: 421918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12341.385623] LNetError: 421918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12342.443796] LNet: Removed LNI 192.168.201.111@tcp [12343.268995] Key type .llcrypt unregistered [12343.271388] Key type ._llcrypt unregistered [12364.242386] Key type ._llcrypt registered [12364.244181] Key type .llcrypt registered [12364.343187] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12365.302734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12365.313362] alg: No test for adler32 (adler32-zlib) [12366.397981] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [12366.632888] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [12368.345521] Key type lgssc registered [12369.210949] Lustre: Echo OBD driver; http://www.lustre.org/ [12375.763703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12375.779442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12377.301376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12381.230747] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12386.789174] LustreError: 423316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12386.797689] LustreError: 423316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12388.620603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12388.944831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12393.353728] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12399.336524] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12404.170243] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12410.073122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12410.311135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12415.458586] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12422.628822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12427.197730] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12434.916540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12434.927280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12434.940509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12435.586779] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.11@tcp (stopping) [12435.872148] Lustre: server umount lustre-MDT0000 complete [12437.990808] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.008317] LustreError: 423315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12438.034307] LustreError: 423315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12439.250897] LustreError: 424051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.11@tcp arrived at 1786444101 with bad export cookie 4362008603783329386 [12439.260823] LustreError: 424051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12441.428850] LustreError: 423295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786444103 with bad export cookie 4362008603783328490 [12441.445173] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12441.451732] LustreError: 423295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12443.105629] LustreError: 424336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12443.111553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12443.139830] LustreError: 424336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12443.183747] Lustre: Skipped 1 previous similar message [12443.201641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12446.179632] LustreError: 423315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12446.183637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12446.210474] LustreError: 423315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12447.753843] Lustre: server umount lustre-OST0000 complete [12451.300601] LustreError: 424043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12451.334926] LustreError: 424043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12458.012316] Lustre: server umount lustre-MDT0001 complete [12474.789634] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12478.037917] Key type lgssc unregistered [12478.419356] LNet: 427005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12478.432367] LNetError: 427005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12478.451263] LNet: Removed LNI 192.168.201.111@tcp [12479.482327] Key type .llcrypt unregistered [12479.485069] Key type ._llcrypt unregistered [12509.459383] Key type ._llcrypt registered [12509.460912] Key type .llcrypt registered [12509.696580] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12512.359429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12512.484364] alg: No test for adler32 (adler32-zlib) [12513.888410] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [12514.104179] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [12515.768404] Key type lgssc registered [12517.541992] Lustre: Echo OBD driver; http://www.lustre.org/ [12527.185190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12527.232341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12528.917253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12535.009640] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12539.378493] LustreError: 428425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12539.400303] LustreError: 428425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12544.482096] LustreError: 428426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12545.067458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12545.373443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12549.977883] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12558.522461] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12566.090816] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12573.385808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12573.808819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12574.863564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12581.209572] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12590.781255] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12595.692227] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12595.703394] Lustre: Skipped 1 previous similar message [12595.710101] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12595.724425] Lustre: Skipped 1 previous similar message [12597.420533] Lustre: server umount lustre-MDT0001 complete [12597.869976] LustreError: 429432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12600.804375] LustreError: 428420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12600.826313] LustreError: 428420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12605.929228] LustreError: 428425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12605.942477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12605.950882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12605.953281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12609.707648] Lustre: server umount lustre-OST0000 complete [12614.149123] Lustre: server umount lustre-MDT0000 complete [12629.957821] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12632.894992] Key type lgssc unregistered [12633.255914] LNet: 432113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12633.265259] LNetError: 432113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12633.295786] LNet: Removed LNI 192.168.201.111@tcp [12634.362647] Key type .llcrypt unregistered [12634.368671] Key type ._llcrypt unregistered [12663.432589] Key type ._llcrypt registered [12663.434723] Key type .llcrypt registered [12663.591836] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [12680.916270] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12682.560724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12682.631118] alg: No test for adler32 (adler32-zlib) [12683.769713] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [12684.193348] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [12685.936636] Key type lgssc registered [12687.385952] Lustre: Echo OBD driver; http://www.lustre.org/ [12732.690802] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12752.655430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12752.728410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12754.086432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12754.106638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12754.208527] Lustre: lustre-MDT0000: new disk, initializing [12754.348277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12754.380140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12760.250894] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12779.002872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12779.329305] Lustre: lustre-OST0000: new disk, initializing [12779.333869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12779.340960] Lustre: Skipped 1 previous similar message [12779.346898] Lustre: 437154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12779.416426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12785.866802] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12787.804805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12787.817783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12787.865282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12798.001527] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12814.361436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12814.514347] Lustre: 436137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12814.572085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12814.773631] Lustre: lustre-MDT0001: new disk, initializing [12814.999450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12815.093783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12815.111096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12820.347847] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12829.946959] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12830.116571] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12834.638955] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12834.874753] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12840.049523] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.11@tcp (stopping) [12841.441229] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12841.449705] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12841.457497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12841.472149] Lustre: Skipped 1 previous similar message [12841.497804] Lustre: Skipped 1 previous similar message [12845.128648] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.11@tcp (stopping) [12845.706958] Lustre: server umount lustre-MDT0001 complete [12851.682899] LustreError: 436144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.700174] LustreError: 436144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12856.804728] LustreError: 436143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12856.829362] LustreError: 436143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12861.925676] LustreError: 436147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12861.949389] LustreError: 436147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12864.483517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12864.495115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12864.518572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12867.052293] LustreError: 436148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12872.162442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12872.165366] Lustre: Skipped 1 previous similar message [12872.176953] LustreError: 436147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12877.284080] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12877.535259] Lustre: server umount lustre-OST0000 complete [12887.395744] Lustre: server umount lustre-MDT0000 complete [12910.405814] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12913.328994] Key type lgssc unregistered [12913.736925] LNet: 440932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12913.756942] LNetError: 440932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12914.795841] LNet: Removed LNI 192.168.201.111@tcp [12916.391822] Key type .llcrypt unregistered [12916.393314] Key type ._llcrypt unregistered [12981.732034] Key type ._llcrypt registered [12981.736590] Key type .llcrypt registered [12981.909301] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12982.491578] Key type .llcrypt unregistered [12982.496517] Key type ._llcrypt unregistered [13003.655868] Key type ._llcrypt registered [13003.661855] Key type .llcrypt registered [13003.827351] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13005.013124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13005.172348] alg: No test for adler32 (adler32-zlib) [13006.450893] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [13006.709656] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [13008.472982] Key type lgssc registered [13009.570068] Lustre: Echo OBD driver; http://www.lustre.org/ [13038.152409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13038.209371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13039.515543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13039.541334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13039.641160] Lustre: lustre-MDT0000: new disk, initializing [13039.765129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13039.810356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13044.692952] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13056.194593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13056.371627] Lustre: 444316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13056.451929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13056.458729] Lustre: Skipped 1 previous similar message [13056.583623] Lustre: lustre-MDT0001: new disk, initializing [13056.696639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13056.745478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13056.770367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13062.516249] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13072.543703] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13078.161744] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13085.640625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13086.040139] Lustre: lustre-OST0000: new disk, initializing [13086.048930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13086.059351] Lustre: 446286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13086.218445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13093.379222] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13095.531194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13095.546699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13095.714321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13103.332172] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13111.278366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13111.289439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13116.388368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13116.400201] Lustre: Skipped 4 previous similar messages [13117.953028] Lustre: server umount lustre-MDT0000 complete [13123.279363] LustreError: 445123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786444785 with bad export cookie 3511644509040850022 [13123.282469] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13123.287578] LustreError: 445123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13124.146470] Lustre: server umount lustre-MDT0001 complete [13138.421414] Lustre: server umount lustre-OST0000 complete [13150.422316] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:40:10 (1786444810) [13158.060697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13158.488448] LustreError: 448315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13158.505851] LustreError: 448315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13158.581628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13163.834489] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13164.007253] LustreError: 448316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.100570] LustreError: 448315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13172.920793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13173.218292] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13173.437269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13178.212481] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13185.836123] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13192.338569] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13214.450590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13214.754885] Lustre: Found index 0 for lustre-OST0000, updating log [13214.980046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13222.743272] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13229.303738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13230.069996] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13234.660581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13234.668633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13234.672264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13234.712548] Lustre: Skipped 3 previous similar messages [13236.505855] Lustre: server umount lustre-OST0000 complete [13255.147566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13255.166881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13255.183430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13255.185852] Lustre: Skipped 3 previous similar messages [13257.201251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13258.781692] Lustre: server umount lustre-MDT0000 complete [13262.306226] LustreError: 448312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13262.338992] LustreError: 448312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13262.386628] LustreError: 448296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786444924 with bad export cookie 3511644509040851415 [13262.400400] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13262.785588] Lustre: server umount lustre-MDT0001 complete [13271.914969] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:42:12 (1786444932) [13282.272158] LNet: 1 local NIs in recovery (showing 1): 192.168.201.111@tcp [13878.165351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13878.413710] LustreError: 454722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13878.465246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13881.086602] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13883.874176] LustreError: 454723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13887.127734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13887.413385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13890.589572] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13895.772288] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13899.634857] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13917.291695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13917.491969] Lustre: Found index 0 for lustre-OST0000, updating log [13917.654653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13922.176673] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13927.406680] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13928.797161] Lustre: server umount lustre-OST0000 complete [13933.537922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13933.542365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13933.550202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13933.552957] Lustre: Skipped 2 previous similar messages [13934.560289] LNet: 1 local NIs in recovery (showing 1): 192.168.201.111@tcp [13934.561839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13934.565276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13936.610533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13936.620192] Lustre: Skipped 4 previous similar messages [13940.706434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13940.711021] Lustre: Skipped 6 previous similar messages [13945.312208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13945.410979] Lustre: server umount lustre-MDT0000 complete [13945.826285] LustreError: 456539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13945.831445] LustreError: 456539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13947.710783] LustreError: 454704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786445609 with bad export cookie 3511644509040852500 [13947.717140] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13947.719478] LustreError: 454704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13947.937873] Lustre: server umount lustre-MDT0001 complete [13971.271977] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [13977.335369] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13999.119512] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:54:20 (1786445660) [14010.990633] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [14012.968605] Key type lgssc unregistered [14013.183771] LNet: 461335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14013.188381] LNetError: 461335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14141.223628] LNet: Removed LNI 192.168.201.111@tcp [14141.725496] Key type .llcrypt unregistered [14141.727234] Key type ._llcrypt unregistered [14151.036882] Key type ._llcrypt registered [14151.038746] Key type .llcrypt registered [14151.105359] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [14151.620826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14151.635545] alg: No test for adler32 (adler32-zlib) [14152.581754] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [14152.746979] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [14154.368363] Key type lgssc registered [14155.115742] Lustre: Echo OBD driver; http://www.lustre.org/ [14157.775854] LNet: Added LNI 192.168.201.111@tcp100 [8/256/0/180] [14158.439019] LNet: Added LNI 192.168.201.111@tcp101 [8/256/0/180] [14159.015366] LNet: Added LNI 192.168.201.111@tcp102 [8/256/0/180] [14159.542646] LNet: Added LNI 192.168.201.111@tcp103 [8/256/0/180] [14160.009441] LNet: Added LNI 192.168.201.111@tcp104 [8/256/0/180] [14160.492133] LNet: Added LNI 192.168.201.111@tcp105 [8/256/0/180] [14160.940636] LNet: Added LNI 192.168.201.111@tcp106 [8/256/0/180] [14161.418332] LNet: Added LNI 192.168.201.111@tcp107 [8/256/0/180] [14161.911051] LNet: Added LNI 192.168.201.111@tcp108 [8/256/0/180] [14162.387707] LNet: Added LNI 192.168.201.111@tcp109 [8/256/0/180] [14162.925684] LNet: Added LNI 192.168.201.111@tcp110 [8/256/0/180] [14163.367671] LNet: Added LNI 192.168.201.111@tcp111 [8/256/0/180] [14163.899535] LNet: Added LNI 192.168.201.111@tcp112 [8/256/0/180] [14164.445626] LNet: Added LNI 192.168.201.111@tcp113 [8/256/0/180] [14165.020340] LNet: Added LNI 192.168.201.111@tcp114 [8/256/0/180] [14165.670984] LNet: Added LNI 192.168.201.111@tcp115 [8/256/0/180] [14166.263091] LNet: Added LNI 192.168.201.111@tcp116 [8/256/0/180] [14166.829884] LNet: Added LNI 192.168.201.111@tcp117 [8/256/0/180] [14167.321597] LNet: Added LNI 192.168.201.111@tcp118 [8/256/0/180] [14167.827852] LNet: Added LNI 192.168.201.111@tcp119 [8/256/0/180] [14168.445514] LNet: Added LNI 192.168.201.111@tcp120 [8/256/0/180] [14168.874222] LNet: Added LNI 192.168.201.111@tcp121 [8/256/0/180] [14169.336891] LNet: Added LNI 192.168.201.111@tcp122 [8/256/0/180] [14169.737241] LNet: Added LNI 192.168.201.111@tcp123 [8/256/0/180] [14170.215369] LNet: Added LNI 192.168.201.111@tcp124 [8/256/0/180] [14170.733579] LNet: Added LNI 192.168.201.111@tcp125 [8/256/0/180] [14171.272858] LNet: Added LNI 192.168.201.111@tcp126 [8/256/0/180] [14171.801101] LNet: Added LNI 192.168.201.111@tcp127 [8/256/0/180] [14172.303665] LNet: Added LNI 192.168.201.111@tcp128 [8/256/0/180] [14172.767932] LNet: Added LNI 192.168.201.111@tcp129 [8/256/0/180] [14173.203049] LNet: Added LNI 192.168.201.111@tcp130 [8/256/0/180] [14173.652249] LNet: Added LNI 192.168.201.111@tcp131 [8/256/0/180] [14174.059309] LNet: Added LNI 192.168.201.111@tcp132 [8/256/0/180] [14174.535433] LNet: Added LNI 192.168.201.111@tcp133 [8/256/0/180] [14174.979878] LNet: Added LNI 192.168.201.111@tcp134 [8/256/0/180] [14175.449176] LNet: Added LNI 192.168.201.111@tcp135 [8/256/0/180] [14179.244669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14179.262889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14180.342305] Lustre: 464344:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.111@tcp: can't add NID 192.168.201.111@tcp130: rc = -75 [14180.414834] LNetError: 461807:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [14180.423684] LNetError: Protocol error connecting to 192.168.201.111@tcp122 at host 192.168.201.111:988: is it running a compatible version of Lustre? [14180.433446] LNetError: Skipped 4 previous similar messages [14180.481093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14180.505363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14180.555689] Lustre: lustre-MDT0000: new disk, initializing [14180.608396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14180.622642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14181.419044] LNet: 2 local NIs in recovery (showing 2): 192.168.201.111@tcp129, 192.168.201.111@tcp122 [14181.421863] LNetError: 461808:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [14181.443018] LNetError: 461808:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14181.450332] LNetError: Protocol error connecting to 192.168.201.111@tcp111 at host 192.168.201.111:988: is it running a compatible version of Lustre? [14181.454571] LNetError: Skipped 1 previous similar message [14183.304261] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [14190.693129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14190.804378] Lustre: 464361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14190.845286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14190.847470] Lustre: Skipped 1 previous similar message [14190.893935] Lustre: lustre-MDT0001: new disk, initializing [14190.913119] Lustre: 465161:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.111@tcp: can't add NID 192.168.201.111@tcp130: rc = -75 [14190.916715] Lustre: 465161:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14190.991860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14191.025207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14191.031549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14193.164184] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [14196.230123] Lustre: 465604:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.111@tcp: can't add NID 192.168.201.111@tcp130: rc = -75 [14196.234539] Lustre: 465604:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14198.642930] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14201.745703] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14205.758549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14205.903847] Lustre: lustre-OST0000: new disk, initializing [14205.907321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14205.913480] Lustre: 466332:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14205.975644] Lustre: 466338:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.111@tcp: can't add NID 192.168.201.111@tcp130: rc = -75 [14205.979297] Lustre: 466338:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14206.003894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14209.629591] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [14210.081192] LNetError: 464367:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [14210.086167] LNetError: Protocol error connecting to 192.168.201.111@tcp119 at host 192.168.201.111:988: is it running a compatible version of Lustre? [14215.173474] Lustre: 467009:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.111@tcp: can't add NID 192.168.201.111@tcp130: rc = -75 [14215.178549] Lustre: 467009:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14215.222282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14215.226916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14215.303667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14215.358614] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14216.226670] LNetError: 461807:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [14216.236609] LNetError: Protocol error connecting to 192.168.201.111@tcp123 at host 192.168.201.111:988: is it running a compatible version of Lustre? [14220.259440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14220.267771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14220.276975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14220.769989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14220.775535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14220.782186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14221.346674] LNetError: 464376:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [14221.356694] LNetError: 464376:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14221.365949] LNetError: Protocol error connecting to 192.168.201.111@tcp124 at host 192.168.201.111:988: is it running a compatible version of Lustre? [14221.371347] LNetError: Skipped 2 previous similar messages [14221.660531] Lustre: server umount lustre-OST0000 complete [14225.893122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14225.898533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14225.899724] Lustre: Skipped 1 previous similar message [14225.906936] Lustre: Skipped 1 previous similar message [14229.539540] LNetError: 461808:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [14229.550170] LNetError: 461808:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14229.556861] LNetError: Protocol error connecting to 192.168.201.111@tcp123 at host 192.168.201.111:988: is it running a compatible version of Lustre? [14229.565182] LNetError: Skipped 2 previous similar messages [14231.012649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14231.018489] Lustre: Skipped 1 previous similar message [14236.130552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14236.132814] Lustre: Skipped 1 previous similar message [14238.176361] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14238.306295] Lustre: server umount lustre-MDT0000 complete [14240.748114] LustreError: 464353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786445902 with bad export cookie 4564592941057675327 [14240.749505] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14240.753525] LustreError: 464353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14240.975362] Lustre: server umount lustre-MDT0001 complete [14247.240271] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [14247.970431] LNetError: 464373:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [14247.975847] LNetError: 464373:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [14247.982405] LNetError: Protocol error connecting to 192.168.201.111@tcp124 at host 192.168.201.111:988: is it running a compatible version of Lustre? [14247.994226] LNetError: Skipped 6 previous similar messages [14249.027839] Key type lgssc unregistered [14249.316494] LNet: 468247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14249.321902] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14249.328795] LNet: Removed LNI 192.168.201.111@tcp [14250.336148] LNet: Removed LNI 192.168.201.111@tcp100 [14250.342248] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14250.345493] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14250.351655] LNet: Removed LNI 192.168.201.111@tcp101 [14251.360243] LNet: Removed LNI 192.168.201.111@tcp102 [14251.364160] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14251.369744] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14251.377415] LNet: Removed LNI 192.168.201.111@tcp103 [14252.386083] LNet: Removed LNI 192.168.201.111@tcp104 [14253.408425] LNet: Removed LNI 192.168.201.111@tcp105 [14253.410409] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14253.420558] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14253.429623] LNet: Removed LNI 192.168.201.111@tcp106 [14254.496154] LNet: Removed LNI 192.168.201.111@tcp107 [14255.520145] LNet: Removed LNI 192.168.201.111@tcp108 [14256.544552] LNet: Removed LNI 192.168.201.111@tcp109 [14257.568168] LNet: Removed LNI 192.168.201.111@tcp110 [14257.570142] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14257.579129] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14257.587602] LNet: Removed LNI 192.168.201.111@tcp111 [14258.656140] LNet: Removed LNI 192.168.201.111@tcp112 [14259.680327] LNet: Removed LNI 192.168.201.111@tcp113 [14260.708985] LNet: Removed LNI 192.168.201.111@tcp114 [14261.728229] LNet: Removed LNI 192.168.201.111@tcp115 [14262.754182] LNet: Removed LNI 192.168.201.111@tcp116 [14263.776161] LNet: Removed LNI 192.168.201.111@tcp117 [14264.800446] LNet: Removed LNI 192.168.201.111@tcp118 [14265.824180] LNet: Removed LNI 192.168.201.111@tcp119 [14265.825626] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14265.828426] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14265.834720] LNet: Removed LNI 192.168.201.111@tcp120 [14266.848187] LNet: Removed LNI 192.168.201.111@tcp121 [14267.872127] LNet: Removed LNI 192.168.201.111@tcp122 [14268.896151] LNet: Removed LNI 192.168.201.111@tcp123 [14269.920168] LNet: Removed LNI 192.168.201.111@tcp124 [14270.948789] LNet: Removed LNI 192.168.201.111@tcp125 [14271.969220] LNet: Removed LNI 192.168.201.111@tcp126 [14272.992143] LNet: Removed LNI 192.168.201.111@tcp127 [14274.016135] LNet: Removed LNI 192.168.201.111@tcp128 [14275.040151] LNet: Removed LNI 192.168.201.111@tcp129 [14275.042852] LNet: Removed LNI 192.168.201.111@tcp130 [14275.044710] LNet: Removed LNI 192.168.201.111@tcp131 [14275.051052] LNet: Removed LNI 192.168.201.111@tcp132 [14275.052461] LNet: Removed LNI 192.168.201.111@tcp133 [14275.054138] LNet: Removed LNI 192.168.201.111@tcp134 [14275.062946] LNet: Removed LNI 192.168.201.111@tcp135 [14275.488554] Key type .llcrypt unregistered [14275.489780] Key type ._llcrypt unregistered [14287.955693] Key type ._llcrypt registered [14287.957739] Key type .llcrypt registered [14288.028368] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [14296.841155] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [14297.531717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14297.552155] alg: No test for adler32 (adler32-zlib) [14298.488175] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [14298.652953] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [14300.320922] Key type lgssc registered [14301.019666] Lustre: Echo OBD driver; http://www.lustre.org/ [14320.060231] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:59:41 (1786445981) [14329.783609] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [14331.349433] Key type lgssc unregistered [14331.519295] LNet: 472312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14331.523916] LNetError: 472312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14331.537693] LNet: Removed LNI 192.168.201.111@tcp [14331.996727] Key type .llcrypt unregistered [14332.001299] Key type ._llcrypt unregistered [14341.659833] Key type ._llcrypt registered [14341.661471] Key type .llcrypt registered [14341.724935] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [14342.332642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14342.367486] alg: No test for adler32 (adler32-zlib) [14343.263954] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [14343.400392] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [14345.024197] Key type lgssc registered [14345.561687] Lustre: Echo OBD driver; http://www.lustre.org/ [14350.414652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14350.426789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14351.594794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14351.617459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14351.666365] Lustre: lustre-MDT0000: new disk, initializing [14351.743689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14351.756241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14354.658360] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [14365.629098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14365.733842] Lustre: 473502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14365.799931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14365.802759] Lustre: Skipped 1 previous similar message [14365.912527] Lustre: lustre-MDT0001: new disk, initializing [14366.016135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14366.050136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14366.068352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14370.853113] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [14390.600988] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14404.960580] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14419.341522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14419.801506] Lustre: lustre-OST0000: new disk, initializing [14419.803932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14419.813506] Lustre: 475484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14419.872242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14427.504312] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [14429.714029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14429.725895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14429.835609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14437.097645] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14440.433829] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing lnet_if_list [14446.248373] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14446.269045] LNet: Added LNI 192.168.201.111@tcp10 [8/256/0/180] [14453.081487] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14453.158708] LNet: Added LNI 192.168.201.111@tcp11 [8/256/0/180] [14458.738652] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14458.760734] LNet: Added LNI 192.168.201.111@tcp12 [8/256/0/180] [14465.293376] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14465.321093] LNet: Added LNI 192.168.201.111@tcp13 [8/256/0/180] [14470.349662] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14470.397652] LNet: Added LNI 192.168.201.111@tcp14 [8/256/0/180] [14474.998250] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14475.050576] LNet: Added LNI 192.168.201.111@tcp15 [8/256/0/180] [14480.435862] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14480.488150] LNet: Added LNI 192.168.201.111@tcp16 [8/256/0/180] [14486.052230] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14486.117358] LNet: Added LNI 192.168.201.111@tcp17 [8/256/0/180] [14491.521986] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14491.568543] LNet: Added LNI 192.168.201.111@tcp18 [8/256/0/180] [14496.776574] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14496.809754] LNet: Added LNI 192.168.201.111@tcp19 [8/256/0/180] [14502.388154] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14502.411975] LNet: Added LNI 192.168.201.111@tcp20 [8/256/0/180] [14508.918786] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14508.939340] LNet: Added LNI 192.168.201.111@tcp21 [8/256/0/180] [14514.914804] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14514.962697] LNet: Added LNI 192.168.201.111@tcp22 [8/256/0/180] [14520.656227] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14520.687978] LNet: Added LNI 192.168.201.111@tcp23 [8/256/0/180] [14526.948870] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14526.995120] LNet: Added LNI 192.168.201.111@tcp24 [8/256/0/180] [14533.170771] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14533.204895] LNet: Added LNI 192.168.201.111@tcp25 [8/256/0/180] [14538.734647] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14538.761934] LNet: Added LNI 192.168.201.111@tcp26 [8/256/0/180] [14545.862253] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14545.903330] LNet: Added LNI 192.168.201.111@tcp27 [8/256/0/180] [14553.501245] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14553.548434] LNet: Added LNI 192.168.201.111@tcp28 [8/256/0/180] [14561.353182] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14561.395102] LNet: Added LNI 192.168.201.111@tcp29 [8/256/0/180] [14568.009361] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14568.024672] LNet: Added LNI 192.168.201.111@tcp30 [8/256/0/180] [14574.342800] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14574.371235] LNet: Added LNI 192.168.201.111@tcp31 [8/256/0/180] [14581.443579] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14581.458881] LNet: Added LNI 192.168.201.111@tcp32 [8/256/0/180] [14589.890453] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14589.926384] LNet: Added LNI 192.168.201.111@tcp33 [8/256/0/180] [14598.124181] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14598.151072] LNet: Added LNI 192.168.201.111@tcp34 [8/256/0/180] [14605.608918] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14605.641533] LNet: Added LNI 192.168.201.111@tcp35 [8/256/0/180] [14612.833478] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14612.904958] LNet: Added LNI 192.168.201.111@tcp36 [8/256/0/180] [14620.682618] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14620.710910] LNet: Added LNI 192.168.201.111@tcp37 [8/256/0/180] [14629.515849] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14629.566574] LNet: Added LNI 192.168.201.111@tcp38 [8/256/0/180] [14638.253875] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14638.306447] LNet: Added LNI 192.168.201.111@tcp39 [8/256/0/180] [14646.517615] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14646.572082] LNet: Added LNI 192.168.201.111@tcp40 [8/256/0/180] [14655.296294] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14655.363417] LNet: Added LNI 192.168.201.111@tcp41 [8/256/0/180] [14664.460781] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14664.477549] LNet: Added LNI 192.168.201.111@tcp42 [8/256/0/180] [14674.608380] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14674.635056] LNet: Added LNI 192.168.201.111@tcp43 [8/256/0/180] [14684.278870] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14684.308199] LNet: Added LNI 192.168.201.111@tcp44 [8/256/0/180] [14695.581150] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14695.645911] LNet: Added LNI 192.168.201.111@tcp45 [8/256/0/180] [14706.180597] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14706.202900] LNet: Added LNI 192.168.201.111@tcp46 [8/256/0/180] [14715.925188] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14716.011939] LNet: Added LNI 192.168.201.111@tcp47 [8/256/0/180] [14724.505831] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14724.538745] LNet: Added LNI 192.168.201.111@tcp48 [8/256/0/180] [14733.913972] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14733.948681] LNet: Added LNI 192.168.201.111@tcp49 [8/256/0/180] [14744.832722] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14744.883506] LNet: Added LNI 192.168.201.111@tcp50 [8/256/0/180] [14768.100076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14768.103578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14768.120695] Lustre: Skipped 1 previous similar message [14773.171395] Lustre: server umount lustre-MDT0000 complete [14777.305634] LustreError: 473495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786446439 with bad export cookie 17912848555187668196 [14777.319235] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14777.322423] LustreError: 473495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14777.674316] Lustre: server umount lustre-MDT0001 complete [14794.721447] Lustre: 472777:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786446440/real 1786446440] req@00000000237c604f x1873224406327552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786446456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14794.756724] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14794.765526] Lustre: Skipped 1 previous similar message [14798.044165] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [14798.511512] Lustre: 472778:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786446444/real 1786446444] req@00000000842f2997 x1873224406327680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786446460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14802.912211] Lustre: 472775:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786446449/real 1786446449] req@000000003d258305 x1873224406328064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786446465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14802.953222] Lustre: 472775:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14807.253423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14808.032118] Lustre: 472777:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786446454/real 1786446454] req@000000008710f0db x1873224406328192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786446470 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14822.504927] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf89738dac7548f64 [14822.518735] Lustre: MGC192.168.201.111@tcp: Connection restored to 0@lo (at 0@lo) [14822.766679] LustreError: 475477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14822.784863] LustreError: 475477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14822.843431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14823.841061] LustreError: 475478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14823.847524] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14824.868129] LustreError: 475477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14828.511935] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [14830.064290] LustreError: 475478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14835.175682] LustreError: 475477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14838.229872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14838.656569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14839.679106] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14843.812892] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [14852.717905] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14859.197280] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14879.201670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14879.212885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14879.227272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14879.239390] Lustre: Skipped 2 previous similar messages [14879.718342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14880.740106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14880.752425] Lustre: Skipped 2 previous similar messages [14885.566866] Lustre: server umount lustre-OST0000 complete [14889.957410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14889.960367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14889.989889] Lustre: Skipped 1 previous similar message [14889.999546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14896.016800] Lustre: server umount lustre-MDT0000 complete [14896.099231] LustreError: 499733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14896.120422] LustreError: 499733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14899.711395] LustreError: 473494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786446561 with bad export cookie 17912848555187670884 [14899.723604] LustreError: 473494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14899.724758] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14900.356623] Lustre: server umount lustre-MDT0001 complete [14912.721072] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [14915.955477] Key type lgssc unregistered [14916.309887] LNet: 502538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14916.318843] LNetError: 502538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14916.323366] LNet: Removed LNI 192.168.201.111@tcp [14916.325654] LNet: Removed LNI 192.168.201.111@tcp10 [14916.330407] LNet: Removed LNI 192.168.201.111@tcp11 [14916.333226] LNet: Removed LNI 192.168.201.111@tcp12 [14916.336383] LNet: Removed LNI 192.168.201.111@tcp13 [14916.338688] LNet: Removed LNI 192.168.201.111@tcp14 [14916.341230] LNet: Removed LNI 192.168.201.111@tcp15 [14916.343638] LNet: Removed LNI 192.168.201.111@tcp16 [14916.345763] LNet: Removed LNI 192.168.201.111@tcp17 [14916.348311] LNet: Removed LNI 192.168.201.111@tcp18 [14916.350369] LNet: Removed LNI 192.168.201.111@tcp19 [14916.352666] LNet: Removed LNI 192.168.201.111@tcp20 [14916.355290] LNet: Removed LNI 192.168.201.111@tcp21 [14916.357526] LNet: Removed LNI 192.168.201.111@tcp22 [14916.363351] LNet: Removed LNI 192.168.201.111@tcp23 [14916.366799] LNet: Removed LNI 192.168.201.111@tcp24 [14916.369678] LNet: Removed LNI 192.168.201.111@tcp25 [14916.375512] LNet: Removed LNI 192.168.201.111@tcp26 [14916.379180] LNet: Removed LNI 192.168.201.111@tcp27 [14916.382502] LNet: Removed LNI 192.168.201.111@tcp28 [14916.388321] LNet: Removed LNI 192.168.201.111@tcp29 [14916.394182] LNet: Removed LNI 192.168.201.111@tcp30 [14916.400406] LNet: Removed LNI 192.168.201.111@tcp31 [14916.404465] LNet: Removed LNI 192.168.201.111@tcp32 [14916.406553] LNet: Removed LNI 192.168.201.111@tcp33 [14916.409446] LNet: Removed LNI 192.168.201.111@tcp34 [14916.412375] LNet: Removed LNI 192.168.201.111@tcp35 [14916.415098] LNet: Removed LNI 192.168.201.111@tcp36 [14916.417716] LNet: Removed LNI 192.168.201.111@tcp37 [14916.420198] LNet: Removed LNI 192.168.201.111@tcp38 [14916.426550] LNet: Removed LNI 192.168.201.111@tcp39 [14916.430662] LNet: Removed LNI 192.168.201.111@tcp40 [14916.433601] LNet: Removed LNI 192.168.201.111@tcp41 [14916.438535] LNet: Removed LNI 192.168.201.111@tcp42 [14916.442798] LNet: Removed LNI 192.168.201.111@tcp43 [14916.445812] LNet: Removed LNI 192.168.201.111@tcp44 [14916.447728] LNet: Removed LNI 192.168.201.111@tcp45 [14916.449739] LNet: Removed LNI 192.168.201.111@tcp46 [14916.451908] LNet: Removed LNI 192.168.201.111@tcp47 [14916.459488] LNet: Removed LNI 192.168.201.111@tcp48 [14916.463419] LNet: Removed LNI 192.168.201.111@tcp49 [14916.471635] LNet: Removed LNI 192.168.201.111@tcp50 [14917.460247] Key type .llcrypt unregistered [14917.463562] Key type ._llcrypt unregistered [14940.408362] Key type ._llcrypt registered [14940.410539] Key type .llcrypt registered [14940.615654] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [14959.351402] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [14961.155969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14961.205706] alg: No test for adler32 (adler32-zlib) [14962.453629] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [14962.704096] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [14964.360922] Key type lgssc registered [14965.804884] Lustre: Echo OBD driver; http://www.lustre.org/ [15000.579909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15000.639301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15001.875703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15001.911318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15001.991832] Lustre: lustre-MDT0000: new disk, initializing [15002.050369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15002.066633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15006.693413] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15017.425612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15017.578844] Lustre: 505737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15017.632315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15017.637449] Lustre: Skipped 1 previous similar message [15017.716280] Lustre: lustre-MDT0001: new disk, initializing [15017.809380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15017.843760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15017.857610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15022.075891] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15030.770261] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15035.644035] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15042.842478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15043.120608] Lustre: *** cfs_fail_loc=90f, val=0*** [15043.224601] Lustre: lustre-OST0000: new disk, initializing [15043.226908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15043.231267] Lustre: 507751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15043.344428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15049.976093] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15051.829372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15051.840532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15051.947945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15059.029940] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15067.106300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15067.112754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15067.133631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15067.621042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15067.623332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15067.642823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15072.749432] Lustre: server umount lustre-OST0000 complete [15077.866743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15077.875695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15077.894993] Lustre: Skipped 1 previous similar message [15077.917316] Lustre: Skipped 1 previous similar message [15083.120561] Lustre: server umount lustre-MDT0000 complete [15086.534496] LustreError: 505728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786446748 with bad export cookie 15601347233015487655 [15086.549867] LustreError: 505728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15086.556649] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15086.839828] Lustre: server umount lustre-MDT0001 complete [15096.949924] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [15099.471445] Key type lgssc unregistered [15099.729874] LNet: 509662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15099.743884] LNetError: 509662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15100.780217] LNet: Removed LNI 192.168.201.111@tcp [15101.912275] Key type .llcrypt unregistered [15101.915365] Key type ._llcrypt unregistered [15122.914841] Key type ._llcrypt registered [15122.923289] Key type .llcrypt registered [15123.075773] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [15124.201037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15124.356441] alg: No test for adler32 (adler32-zlib) [15125.509309] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [15125.825655] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [15127.576639] Key type lgssc registered [15128.912324] Lustre: Echo OBD driver; http://www.lustre.org/ [15136.875633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15136.893703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15138.231846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15142.417739] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15148.517039] LustreError: 510940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15148.530637] LustreError: 510940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15149.538312] LustreError: 510941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15151.159246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15151.678995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15156.239783] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15162.694911] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15168.424749] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15174.716947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15175.046840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15180.301496] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15187.662581] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15198.612102] loop0: detected capacity change from 0 to 204800000 [15198.833141] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15198.845743] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15198.854833] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15204.642868] loop0: detected capacity change from 0 to 204800000 [15204.875030] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15204.884835] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15204.895090] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15210.235550] loop0: detected capacity change from 0 to 204800000 [15215.882676] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15216.013801] Lustre: fs15246-OST0000: new disk, initializing [15216.017091] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15216.022225] Lustre: 514528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15216.070639] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15222.166632] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15230.486570] loop1: detected capacity change from 0 to 204800000 [15235.407397] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15235.539312] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15235.593713] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15235.703619] Lustre: fs15246-MDT0000: new disk, initializing [15235.966873] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15236.009940] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15236.230207] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15237.283694] Lustre: 515651:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000081ae18a9 x1873225227219968/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786446915 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15240.294185] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15246.378791] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15246.390718] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15246.423040] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15259.621161] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15259.651423] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15264.746451] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15265.348633] Lustre: server umount fs15246-MDT0000 complete [15269.866098] LustreError: 510936:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15272.940914] Lustre: server umount fs15246-OST0000 complete [15283.693807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15283.700489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15283.712187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15285.219926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15289.094680] Lustre: server umount lustre-OST0000 complete [15294.945556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15294.955847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15294.965262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15294.970227] Lustre: Skipped 2 previous similar messages [15298.904187] Lustre: server umount lustre-MDT0000 complete [15300.577975] LustreError: 510940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15300.596136] LustreError: 510940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15302.338980] LustreError: 510919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786446964 with bad export cookie 16853371801534065613 [15302.341142] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15302.351306] LustreError: 510919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15302.705877] Lustre: server umount lustre-MDT0001 complete [15312.670690] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [15315.101695] Key type lgssc unregistered [15315.442225] LNet: 518749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15315.452566] LNetError: 518749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15315.465451] LNet: Removed LNI 192.168.201.111@tcp [15316.531725] Key type .llcrypt unregistered [15316.533461] Key type ._llcrypt unregistered [15336.238188] Key type ._llcrypt registered [15336.243381] Key type .llcrypt registered [15336.416282] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [15337.951975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15337.979839] alg: No test for adler32 (adler32-zlib) [15339.146175] Lustre: Lustre: Build Version: 2.17.56_51_gd63417b [15339.508519] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [15341.256179] Key type lgssc registered [15342.857338] Lustre: Echo OBD driver; http://www.lustre.org/ [15385.034356] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [15396.747766] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [15429.339405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15429.363054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15430.602946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15430.635518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15430.719465] Lustre: lustre-MDT0000: new disk, initializing [15430.835250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15430.849763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15435.579432] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15445.704163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15445.852302] Lustre: 523216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15445.904282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15445.908129] Lustre: Skipped 1 previous similar message [15445.975841] Lustre: lustre-MDT0001: new disk, initializing [15446.088787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15446.132589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15446.146141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15451.547659] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15462.039806] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15468.017462] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15474.556575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15474.837982] Lustre: lustre-OST0000: new disk, initializing [15474.843569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15474.854873] Lustre: 525186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15474.943589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15481.457295] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [15482.458947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15482.472726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15482.651811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15491.040560] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15495.907777] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15496.220377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15500.943119] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15501.148193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15503.330609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15503.334614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15503.343686] LustreError: Skipped 1 previous similar message [15503.344760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15503.376583] Lustre: Skipped 1 previous similar message [15508.450718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15508.456945] Lustre: Skipped 1 previous similar message [15513.572426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15513.583144] Lustre: Skipped 1 previous similar message [15517.153625] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15517.333972] Lustre: server umount lustre-OST0000 complete [15522.785945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15522.794677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15522.809095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15523.830378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15527.449579] Lustre: server umount lustre-MDT0000 complete [15530.313894] LustreError: 523209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786447192 with bad export cookie 10235938909243987134 [15530.320517] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15530.324353] LustreError: 523209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15530.625924] Lustre: server umount lustre-MDT0001 complete [15549.535746] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [15559.387895] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [15591.833571] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15304 sec ========= 07:20:52 (1786447252) [15593.721647] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:20:54 (1786447254) === [15597.674524] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:20:58 (1786447258) === [15635.021979] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [15637.550637] Key type lgssc unregistered [15637.772485] LNet: 531833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15637.776198] LNetError: 531833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15637.790320] LNet: Removed LNI 192.168.201.111@tcp [15638.542740] Key type .llcrypt unregistered [15638.545177] Key type ._llcrypt unregistered