[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 720800806 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002000] LSM: Security Framework initializing [ 0.002000] Yama: becoming mindful. [ 0.002000] SELinux: Initializing. [ 0.002000] *** VALIDATE selinux *** [ 0.002000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.002000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.002000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] *** VALIDATE tmpfs *** [ 0.002000] *** VALIDATE proc *** [ 0.002000] *** VALIDATE cgroup *** [ 0.002000] *** VALIDATE cgroup2 *** [ 0.002000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.002000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.002000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.002000] Spectre V2 : User space: Vulnerable [ 0.002000] Speculative Store Bypass: Vulnerable [ 0.002000] debug: unmapping init [mem 0xffffffff95859000-0xffffffff95860fff] [ 0.002000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.002726] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.003026] ... version: 2 [ 0.004015] ... bit width: 48 [ 0.005013] ... generic registers: 4 [ 0.006016] ... value mask: 0000ffffffffffff [ 0.007017] ... max period: 00007fffffffffff [ 0.008015] ... fixed-purpose events: 3 [ 0.009013] ... event mask: 000000070000000f [ 0.010328] rcu: Hierarchical SRCU implementation. [ 0.012869] smp: Bringing up secondary CPUs ... [ 0.013580] x86: Booting SMP configuration: [ 0.014038] .... node #0, CPUs: #1 #2 #3 [ 0.027658] smp: Brought up 1 node, 4 CPUs [ 0.029020] smpboot: Max logical packages: 1 [ 0.030016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.057812] node 0 deferred pages initialised in 25ms [ 0.063259] devtmpfs: initialized [ 0.064388] x86/mm: Memory block size: 128MB [ 0.067224] gcov: version magic: 0x41383552 [ 0.074073] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.075362] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.076316] pinctrl core: initialized pinctrl subsystem [ 0.077318] [ 0.078010] ************************************************************* [ 0.079021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.080018] ** ** [ 0.081018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.082018] ** ** [ 0.083013] ** This means that this kernel is built to expose internal ** [ 0.084014] ** IOMMU data structures, which may compromise security on ** [ 0.085018] ** your system. ** [ 0.086018] ** ** [ 0.087021] ** If you see this message and you are not debugging the ** [ 0.088017] ** kernel, report this immediately to your vendor! ** [ 0.089021] ** ** [ 0.090020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.091017] ************************************************************* [ 0.092924] NET: Registered protocol family 16 [ 0.093365] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.094078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.095056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.096657] cpuidle: using governor menu [ 0.098749] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.102019] PCI: Using configuration type 1 for base access [ 0.104190] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.116365] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.120019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.123734] cryptd: max_cpu_qlen set to 1000 [ 0.124581] ACPI: Added _OSI(Module Device) [ 0.125000] ACPI: Added _OSI(Processor Device) [ 0.125000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.128020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.135268] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.146304] ACPI: Interpreter enabled [ 0.147071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.148015] ACPI: Using IOAPIC for interrupt routing [ 0.149251] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.150551] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.164556] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.165249] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.166024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.167107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.169711] acpiphp: Slot [2] registered [ 0.170231] acpiphp: Slot [5] registered [ 0.171398] acpiphp: Slot [6] registered [ 0.172345] acpiphp: Slot [7] registered [ 0.173200] acpiphp: Slot [8] registered [ 0.174175] acpiphp: Slot [9] registered [ 0.175179] acpiphp: Slot [10] registered [ 0.176183] acpiphp: Slot [3] registered [ 0.177168] acpiphp: Slot [4] registered [ 0.178130] acpiphp: Slot [11] registered [ 0.179143] acpiphp: Slot [12] registered [ 0.180133] acpiphp: Slot [13] registered [ 0.181132] acpiphp: Slot [14] registered [ 0.182123] acpiphp: Slot [15] registered [ 0.183103] acpiphp: Slot [16] registered [ 0.184515] acpiphp: Slot [17] registered [ 0.185268] acpiphp: Slot [18] registered [ 0.186121] acpiphp: Slot [19] registered [ 0.187118] acpiphp: Slot [20] registered [ 0.188219] acpiphp: Slot [21] registered [ 0.189129] acpiphp: Slot [22] registered [ 0.190151] acpiphp: Slot [23] registered [ 0.191123] acpiphp: Slot [24] registered [ 0.192172] acpiphp: Slot [25] registered [ 0.193121] acpiphp: Slot [26] registered [ 0.194121] acpiphp: Slot [27] registered [ 0.195130] acpiphp: Slot [28] registered [ 0.196238] acpiphp: Slot [29] registered [ 0.197239] acpiphp: Slot [30] registered [ 0.198428] acpiphp: Slot [31] registered [ 0.199073] PCI host bridge to bus 0000:00 [ 0.200026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.201042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.202027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.203027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.204027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.205042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.206283] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.208375] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.210255] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.215696] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.218055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.219020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.220016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.221013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.222510] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.224054] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.225041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.226784] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.229015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.238014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.240014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.244000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.246019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.249540] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.256020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.265478] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.268038] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.271028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.278036] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.283882] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.286019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.289023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.296025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.302168] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.305025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.308038] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.315023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.320866] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.323037] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.326026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.333026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.339180] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.342025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.345027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.352025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.358812] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.359403] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.360337] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.361395] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.362239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.459099] iommu: Default domain type: Passthrough [ 0.460604] SCSI subsystem initialized [ 0.461130] ACPI: bus type USB registered [ 0.462133] usbcore: registered new interface driver usbfs [ 0.463084] usbcore: registered new interface driver hub [ 0.464168] usbcore: registered new device driver usb [ 0.465296] pps_core: LinuxPPS API ver. 1 registered [ 0.466013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.467058] PTP clock support registered [ 0.469059] EDAC MC: Ver: 3.0.0 [ 0.639551] PCI: Using ACPI for IRQ routing [ 0.644470] NetLabel: Initializing [ 0.647014] NetLabel: domain hash size = 128 [ 0.649016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.651166] NetLabel: unlabeled traffic allowed by default [ 0.654276] vgaarb: loaded [ 0.656475] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.658016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.665658] clocksource: Switched to clocksource kvm-clock [ 0.790350] VFS: Disk quotas dquot_6.6.0 [ 0.791852] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.794515] *** VALIDATE ramfs *** [ 0.795954] *** VALIDATE hugetlbfs *** [ 0.797756] pnp: PnP ACPI init [ 0.800512] pnp: PnP ACPI: found 6 devices [ 0.817794] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.821037] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.823175] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.825415] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.827719] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.830299] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.833244] NET: Registered protocol family 2 [ 0.835707] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.840698] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.844725] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.849931] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.853174] TCP: Hash tables configured (established 65536 bind 65536) [ 0.855779] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.858125] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.860184] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.862424] NET: Registered protocol family 1 [ 0.864479] RPC: Registered named UNIX socket transport module. [ 0.866223] RPC: Registered udp transport module. [ 0.867387] RPC: Registered tcp transport module. [ 0.868638] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.870319] NET: Registered protocol family 44 [ 0.871593] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.873082] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.874427] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.875985] PCI: CLS 0 bytes, default 64 [ 0.877850] Unpacking initramfs... [ 2.719619] debug: unmapping init [mem 0xffff94c1bcc54000-0xffff94c1bffbffff] [ 2.747515] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.750012] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.754299] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.744527] Initialise system trusted keyrings [ 3.746223] Key type blacklist registered [ 3.748198] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.758305] zbud: loaded [ 3.761581] *** VALIDATE nfs *** [ 3.762935] *** VALIDATE nfs4 *** [ 3.764557] pstore: using deflate compression [ 3.768089] Platform Keyring initialized [ 3.891225] NET: Registered protocol family 38 [ 3.893304] Key type asymmetric registered [ 3.894878] Asymmetric key parser 'x509' registered [ 3.896928] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.900704] io scheduler mq-deadline registered [ 3.902523] io scheduler kyber registered [ 3.904500] io scheduler bfq registered [ 3.906170] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.909320] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.912454] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.916600] ACPI: Power Button [PWRF] [ 3.924146] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.933262] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.981505] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.139801] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.154068] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.190277] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.225036] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.229829] Non-volatile memory driver v1.3 [ 4.231485] Linux agpgart interface v0.103 [ 4.275447] virtio_blk virtio1: [vda] 150080 512-byte logical blocks (76.8 MB/73.3 MiB) [ 4.328273] vda: detected capacity change from 0 to 76840960 [ 4.368788] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.370636] vdb: detected capacity change from 0 to 1073741824 [ 4.382202] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.384175] vdc: detected capacity change from 0 to 2621440000 [ 4.396365] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.399200] vdd: detected capacity change from 0 to 2621440000 [ 4.413708] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.416885] vde: detected capacity change from 0 to 4294967296 [ 4.441345] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.446335] vdf: detected capacity change from 0 to 4294967296 [ 4.461432] libphy: Fixed MDIO Bus: probed [ 4.467839] usbcore: registered new interface driver usbserial_generic [ 4.470864] usbserial: USB Serial support registered for generic [ 4.473021] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.480520] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.482808] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.486544] mousedev: PS/2 mouse device common for all mice [ 4.489750] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.493742] rtc_cmos 00:05: RTC can wake from S4 [ 4.498984] rtc_cmos 00:05: registered as rtc0 [ 4.501490] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.505989] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.509524] intel_pstate: CPU model not supported [ 4.512817] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.517717] hid: raw HID events driver (C) Jiri Kosina [ 4.520595] usbcore: registered new interface driver usbhid [ 4.522476] usbhid: USB HID core driver [ 4.524328] drop_monitor: Initializing network drop monitor service [ 4.526796] Initializing XFRM netlink socket [ 4.528922] NET: Registered protocol family 10 [ 4.532159] Segment Routing with IPv6 [ 4.534032] NET: Registered protocol family 17 [ 4.535982] mpls_gso: MPLS GSO support [ 4.541777] RAS: Correctable Errors collector initialized. [ 4.543866] AVX version of gcm_enc/dec engaged. [ 4.545602] AES CTR mode by8 optimization enabled [ 4.636629] sched_clock: Marking stable (4636612055, 0)->(6308922107, -1672310052) [ 4.640586] registered taskstats version 1 [ 4.642750] Loading compiled-in X.509 certificates [ 4.644903] zswap: loaded using pool lzo/zbud [ 4.672567] Key type big_key registered [ 4.688490] Key type encrypted registered [ 4.690353] ima: No TPM chip found, activating TPM-bypass! [ 4.692430] ima: Allocated hash algorithm: sha1 [ 4.694416] ima: No architecture policies found [ 4.696404] evm: Initialising EVM extended attributes: [ 4.698426] evm: security.selinux [ 4.699615] evm: security.ima [ 4.700754] evm: security.capability [ 4.702332] evm: HMAC attrs: 0x1 [ 4.789477] rtc_cmos 00:05: setting system clock to 2026-09-07 03:43:17 UTC (1788752597) [ 4.802338] debug: unmapping init [mem 0xffffffff96803000-0xffffffff969fffff] [ 4.806330] debug: unmapping init [mem 0xffffffff95582000-0xffffffff95858fff] [ 4.816082] Write protecting the kernel read-only data: 28672k [ 4.823277] debug: unmapping init [mem 0xffffffff93c03000-0xffffffff93dfffff] [ 4.828847] debug: unmapping init [mem 0xffffffff94514000-0xffffffff945fffff] [ 4.924870] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.935442] systemd[1]: Detected virtualization kvm. [ 4.937587] systemd[1]: Detected architecture x86-64. [ 4.939938] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.033769] systemd[1]: No hostname configured. [ 5.035747] systemd[1]: Set hostname to . [ 5.038179] random: systemd: uninitialized urandom read (16 bytes read) [ 5.040522] systemd[1]: Initializing machine ID from random generator. [ 5.105069] random: ln: uninitialized urandom read (6 bytes read) [ 5.469832] random: systemd: uninitialized urandom read (16 bytes read) [ 5.474363] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.487381] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.491506] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.496429] device-mapper: uevent: version 1.0.3 [ 7.498707] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.371089] virtio_net virtio0 ens2: renamed from eth0 [ 9.437981] random: fast init done [ 9.822126] scsi host0: ata_piix [ 10.009771] scsi host1: ata_piix [ 10.017216] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.023474] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.340969] random: crng init done [ 15.342832] random: 7 urandom warning(s) missed due to ratelimiting [ 16.271252] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.059237] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.109727] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.320453] SELinux: Disabled at runtime. [ 25.523842] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.546916] systemd[1]: Detected virtualization kvm. [ 25.550871] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.755643] hrtimer: interrupt took 5626352 ns [ 28.054942] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.089592] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.117540] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.126576] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.160680] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.217094] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.269701] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths.[ 29.710972] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 32.272399] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 34.477874] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 34.586452] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 35.170072] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.269059] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (11s / no limit) [** ] A start job is running for Configur…only root support (11s / no limit) [*** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit)[ 42.216265] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (14s / no limit)[ 43.079314] NFS: Registering the id_resolver key type [ 43.081401] Key type id_resolver registered [ 43.086402] Key type id_legacy registered [ **] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg256-server login: [ 107.648506] libcfs: loading out-of-tree module taints kernel. [ 107.684576] Key type ._llcrypt registered [ 107.688842] Key type .llcrypt registered [ 107.854227] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 127.260772] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 128.811215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.834869] alg: No test for adler32 (adler32-zlib) [ 130.294963] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 131.297275] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 133.112370] Key type lgssc registered [ 134.992462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 151.549302] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 195.722989] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 209.861476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 209.887662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.176370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 211.249979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 211.352104] Lustre: lustre-MDT0000: new disk, initializing [ 211.508914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.542319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 217.021113] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.326722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.463720] Lustre: 6531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 232.493666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 232.497466] Lustre: Skipped 1 previous similar message [ 232.610235] Lustre: lustre-MDT0001: new disk, initializing [ 232.694140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.754958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 232.771576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 237.578877] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.905471] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 251.971597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.220531] Lustre: lustre-OST0000: new disk, initializing [ 252.224555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 252.229922] Lustre: 8469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.298115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.657477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 258.675394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 258.756057] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.818517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 273.171839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.340584] Lustre: lustre-OST0001: new disk, initializing [ 273.344995] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 273.351509] Lustre: 9543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 273.406180] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 280.366762] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.663064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 281.687110] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 281.777580] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 292.700653] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 300.359725] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 308.001920] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing check_logdir /tmp/testlogs/ [ 313.198780] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing yml_node [ 318.502503] Lustre: DEBUG MARKER: Client: 2.17.58.43 [ 321.063298] Lustre: DEBUG MARKER: MDS: 2.17.58.43 [ 323.726232] Lustre: DEBUG MARKER: OSS: 2.17.58.43 [ 325.384535] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 23:48:35 EDT 2026 [ 343.047205] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 344.653604] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 360.930446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 360.936995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.974291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.001528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.003345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.468065] Lustre: server umount lustre-MDT0000 complete [ 370.014724] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752962 with bad export cookie 14711482997248026044 [ 370.021310] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 370.031849] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 374.242961] LustreError: 6536:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.245902] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.247987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.247996] Lustre: Skipped 3 previous similar messages [ 374.267340] LustreError: 6536:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 374.329118] Lustre: Skipped 3 previous similar messages [ 375.265158] LustreError: 8477:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.287621] LustreError: 8477:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 376.652209] Lustre: server umount lustre-MDT0001 complete [ 380.460125] Lustre: server umount lustre-OST0000 complete [ 385.032138] Lustre: server umount lustre-OST0001 complete [ 392.426773] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 404.184781] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 444.860039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.159415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 445.227229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 445.320667] Lustre: lustre-MDT0000: new disk, initializing [ 445.461960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.484800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 450.749439] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 462.668903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.841063] Lustre: 15096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.951421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.966753] Lustre: Skipped 1 previous similar message [ 463.102646] Lustre: lustre-MDT0001: new disk, initializing [ 463.223841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.305279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.321110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.865481] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 479.467475] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 485.671807] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 493.262919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 493.541086] Lustre: lustre-OST0000: new disk, initializing [ 493.545411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 493.563470] Lustre: 17066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 493.684957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.501250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 495.517205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 495.679373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 501.185477] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 511.628804] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 517.130647] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 517.359452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 522.713396] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 523.076944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 526.310252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.313647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.335711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.340941] LustreError: Skipped 1 previous similar message [ 526.353475] Lustre: Skipped 4 previous similar messages [ 531.432287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.443740] Lustre: Skipped 1 previous similar message [ 531.729881] Lustre: server umount lustre-OST0000 complete [ 536.554867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.567906] Lustre: Skipped 1 previous similar message [ 541.668504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 541.674347] Lustre: Skipped 3 previous similar messages [ 542.641732] Lustre: server umount lustre-MDT0000 complete [ 545.499185] LustreError: 15891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753138 with bad export cookie 14711482997248027577 [ 545.509191] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.516411] LustreError: 15891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 545.887141] Lustre: server umount lustre-MDT0001 complete [ 550.685122] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:52:21 (1788753141) [ 553.576763] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 555.710284] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:52:25 (1788753145) [ 563.073266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.719196] LustreError: 19475:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.789416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.519914] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 568.813989] LustreError: 19476:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.922379] LustreError: 19475:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.632174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.210289] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 592.031339] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 598.588651] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 612.007438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 621.664732] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 632.333138] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 649.703946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 649.718711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.737305] Lustre: Skipped 1 previous similar message [ 649.742490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 654.773359] Lustre: server umount lustre-OST0000 complete [ 667.056291] Lustre: server umount lustre-MDT0000 complete [ 671.203028] LustreError: 19472:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.219691] LustreError: 19472:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 671.931976] LustreError: 19456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753264 with bad export cookie 14711482997248028459 [ 671.934961] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 671.955399] LustreError: 19456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 672.509314] Lustre: server umount lustre-MDT0001 complete [ 689.173423] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 692.798524] Key type lgssc unregistered [ 693.045923] LNet: 23031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.064635] LNetError: 23031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.088618] LNet: Removed LNI 192.168.202.156@tcp [ 694.305159] Key type .llcrypt unregistered [ 694.306469] Key type ._llcrypt unregistered [ 727.311896] Key type ._llcrypt registered [ 727.314464] Key type .llcrypt registered [ 727.504383] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 729.077548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 729.246348] alg: No test for adler32 (adler32-zlib) [ 730.442116] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 730.785862] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 732.482127] Key type lgssc registered [ 733.756146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 744.309056] loop: module loaded [ 744.443698] loop0: detected capacity change from 0 to 204800000 [ 744.782897] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 744.818732] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 745.235177] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 753.271239] loop0: detected capacity change from 0 to 204800000 [ 753.573119] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 753.583345] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 753.592722] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 760.809800] loop0: detected capacity change from 0 to 204800000 [ 768.699250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 768.772730] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.259608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 770.394813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 770.607869] Lustre: lustre-MDT0000: new disk, initializing [ 770.810922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.848947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 776.805523] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 788.003294] loop1: detected capacity change from 0 to 204800000 [ 796.603363] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 797.040786] Lustre: lustre-OST07c6: new disk, initializing [ 797.044899] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 797.048834] Lustre: Skipped 1 previous similar message [ 797.059994] Lustre: 26217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 797.144522] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 805.348228] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 806.573655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 806.580705] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 806.622976] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 811.171479] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 811.180173] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 811.187646] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 811.197132] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 811.206787] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 811.214738] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 826.824868] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 826.828533] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 826.833871] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 826.839388] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 826.845848] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 826.857359] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 832.735565] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 832.747436] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 832.754689] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 832.767448] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.784228] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 832.801306] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.807750] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 832.819088] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.839241] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 832.848990] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.862840] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 832.877492] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.424417] Lustre: 25201:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 838.436526] Lustre: 25201:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 838.443119] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 838.453418] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.466086] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 838.472522] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.480625] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 838.488776] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.495102] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 838.499692] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.504434] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 838.509196] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 846.282221] Lustre: 26224:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 846.291193] Lustre: 26224:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 846.298711] Lustre: 26224:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 846.305056] Lustre: 26224:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 846.312563] Lustre: 26224:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 846.330206] Lustre: 26224:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 846.344381] Lustre: 26224:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 846.353815] Lustre: 26224:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 846.364828] Lustre: 26224:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 846.375444] Lustre: 26224:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 846.381198] Lustre: 26224:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 846.390194] Lustre: 26224:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 857.742673] Lustre: 26224:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 857.755527] Lustre: 26224:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 857.771310] Lustre: 26224:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 857.783593] Lustre: 26224:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 857.796875] Lustre: 26224:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 857.802982] Lustre: 26224:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 857.810609] Lustre: 26224:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 857.817168] Lustre: 26224:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 857.822912] Lustre: 26224:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 857.827637] Lustre: 26224:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 857.832137] Lustre: 26224:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 857.836407] Lustre: 26224:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 903.432600] Lustre: server umount lustre-OST07c6 complete [ 926.470279] Lustre: server umount lustre-MDT0000 complete [ 949.410989] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 953.178394] Key type lgssc unregistered [ 953.581965] LNet: 28774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.607683] LNetError: 28774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.646305] LNet: Removed LNI 192.168.202.156@tcp [ 954.776065] Key type .llcrypt unregistered [ 954.777919] Key type ._llcrypt unregistered [ 977.150827] Key type ._llcrypt registered [ 977.158854] Key type .llcrypt registered [ 977.376717] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 978.862516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.911828] alg: No test for adler32 (adler32-zlib) [ 980.354797] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 980.639438] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 982.329512] Key type lgssc registered [ 984.089625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 993.949686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 993.965221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.745996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.562591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1006.053647] LustreError: 30044:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.086419] LustreError: 30044:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1011.195431] LustreError: 30043:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.026912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.618897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1019.697842] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1028.533683] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1035.942256] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1044.468625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1044.962495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.686208] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1062.541671] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1065.485612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1083.879419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1083.885335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.905958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1085.426117] Lustre: server umount lustre-OST0000 complete [ 1090.529516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1090.540131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.562264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.162282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.173125] Lustre: Skipped 3 previous similar messages [ 1100.488771] Lustre: server umount lustre-MDT0000 complete [ 1101.286514] LustreError: 30038:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1101.313690] LustreError: 30038:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1104.853875] LustreError: 30023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753697 with bad export cookie 17393926810559637657 [ 1104.863626] LustreError: 30023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1104.864582] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.604059] Lustre: server umount lustre-MDT0001 complete [ 1118.782451] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1123.227572] Key type lgssc unregistered [ 1123.663091] LNet: 33800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.673706] LNetError: 33800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.701026] LNet: Removed LNI 192.168.202.156@tcp [ 1124.970054] Key type .llcrypt unregistered [ 1124.972630] Key type ._llcrypt unregistered [ 1146.110463] Key type ._llcrypt registered [ 1146.118136] Key type .llcrypt registered [ 1146.323829] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1147.651267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1147.790385] alg: No test for adler32 (adler32-zlib) [ 1149.120968] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 1149.405914] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1151.088547] Key type lgssc registered [ 1152.142684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1159.896512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1159.906413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.370906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.845022] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1171.939487] LustreError: 35082:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.968593] LustreError: 35082:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1175.843531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.221830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1181.764979] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1189.990523] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1196.389696] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1204.322612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.879817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.984312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1211.618236] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1220.292374] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1235.955114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.959903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1235.978987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1240.449538] Lustre: server umount lustre-OST0000 complete [ 1246.177897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.179978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.201055] Lustre: Skipped 2 previous similar messages [ 1246.214484] Lustre: Skipped 2 previous similar messages [ 1251.199939] Lustre: server umount lustre-MDT0000 complete [ 1251.317567] LustreError: 35810:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.344910] LustreError: 35810:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1254.927349] LustreError: 35061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753847 with bad export cookie 9919854636205346383 [ 1254.940050] LustreError: 35061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1254.942370] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.457782] Lustre: server umount lustre-MDT0001 complete [ 1267.977265] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1271.665041] Key type lgssc unregistered [ 1272.099906] LNet: 38633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.111609] LNetError: 38633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.132752] LNet: Removed LNI 192.168.202.156@tcp [ 1274.103512] Key type .llcrypt unregistered [ 1274.105377] Key type ._llcrypt unregistered [ 1299.977406] Key type ._llcrypt registered [ 1299.982597] Key type .llcrypt registered [ 1300.116424] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1301.853627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1301.921066] alg: No test for adler32 (adler32-zlib) [ 1302.987205] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 1303.251364] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1304.969170] Key type lgssc registered [ 1306.113867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1314.853650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1314.903945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.716927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.569770] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1327.078628] LustreError: 40050:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.091824] LustreError: 40050:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1331.308187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.674066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1336.855859] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1345.000734] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1351.253637] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1358.464623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1358.891924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.001528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1366.299893] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1374.162444] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1387.195573] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1387.493373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.526295] Lustre: Skipped 1 previous similar message [ 1390.824126] Lustre: server umount lustre-MDT0000 complete [ 1392.294549] LustreError: 40046:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.251897] LustreError: 40785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753987 with bad export cookie 9984118409427113000 [ 1395.267801] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1397.406330] LustreError: 40045:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.433946] LustreError: 40045:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 1397.444674] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 1397.450703] Lustre: Skipped 3 previous similar messages [ 1397.737041] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.751083] Lustre: Skipped 1 previous similar message [ 1400.737662] LustreError: 40044:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.737763] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1400.759525] LustreError: 40044:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1400.784115] Lustre: Skipped 1 previous similar message [ 1401.800755] Lustre: server umount lustre-MDT0001 complete [ 1417.203545] Lustre: server umount lustre-OST0000 complete [ 1436.162706] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1439.485877] Key type lgssc unregistered [ 1439.817616] LNet: 43910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.845592] LNetError: 43910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.878996] LNet: Removed LNI 192.168.202.156@tcp [ 1441.733831] Key type .llcrypt unregistered [ 1441.736419] Key type ._llcrypt unregistered [ 1469.153176] Key type ._llcrypt registered [ 1469.155709] Key type .llcrypt registered [ 1469.251744] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1470.736955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1470.928443] alg: No test for adler32 (adler32-zlib) [ 1471.976822] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 1472.318283] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1474.136772] Key type lgssc registered [ 1475.523386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1484.394166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1484.435120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1486.013460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.889295] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1495.528540] LustreError: 45326:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.541284] LustreError: 45326:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1500.645178] LustreError: 45325:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.065592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.461576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1506.807431] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1516.595039] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1521.884467] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1529.871529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1530.287214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.316411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1537.274875] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1545.513724] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1556.965483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1556.970519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.985954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1557.692043] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1557.988971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1558.001687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.933575] Lustre: server umount lustre-OST0000 complete [ 1579.855311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.865487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.886729] Lustre: Skipped 1 previous similar message [ 1579.906621] Lustre: Skipped 3 previous similar messages [ 1582.881731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.885476] Lustre: Skipped 1 previous similar message [ 1588.194543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1594.336109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1594.817478] Lustre: server umount lustre-MDT0000 complete [ 1597.424632] LustreError: 45326:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.456060] LustreError: 45326:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1599.237770] LustreError: 45305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754191 with bad export cookie 17066708113416697502 [ 1599.256059] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1599.258781] LustreError: 45305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1606.021360] Lustre: server umount lustre-MDT0001 complete [ 1618.370836] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1621.469931] Key type lgssc unregistered [ 1621.801699] LNet: 49036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.812057] LNetError: 49036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.824931] LNet: Removed LNI 192.168.202.156@tcp [ 1623.798430] Key type .llcrypt unregistered [ 1623.800645] Key type ._llcrypt unregistered [ 1649.162571] Key type ._llcrypt registered [ 1649.165968] Key type .llcrypt registered [ 1649.290323] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1650.670305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1650.914906] alg: No test for adler32 (adler32-zlib) [ 1652.191577] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 1652.606335] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1654.328300] Key type lgssc registered [ 1655.376442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1664.080692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1664.118381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.586861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.239734] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1675.754603] LustreError: 50448:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1675.774335] LustreError: 50448:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1679.057762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.489385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1685.155642] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1695.650726] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1703.398891] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1711.747153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1712.274807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.075107] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1727.250475] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1735.939943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1744.505401] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1748.520851] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1760.727773] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-07 0h12m30s [ 1762.468945] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1765.046974] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1766.122257] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1766.370172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1766.374726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.483691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.488066] Lustre: Skipped 5 previous similar messages [ 1770.849032] Lustre: server umount lustre-MDT0000 complete [ 1771.168544] LustreError: 50444:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.538813] LustreError: 50444:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.567020] LustreError: 50444:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1775.007550] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1777.315509] LustreError: 50444:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1782.029681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.203606] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1782.521633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.591276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1783.551029] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1787.255294] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1791.041506] Lustre: DEBUG MARKER: Wait for df (36733) ... [ 1792.852221] Lustre: DEBUG MARKER: done [ 1798.123017] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1798.126214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.142753] Lustre: Skipped 2 previous similar messages [ 1798.148711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1799.163392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.646705] Lustre: server umount lustre-OST0000 complete [ 1809.379771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.387802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1809.402869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.416220] Lustre: Skipped 3 previous similar messages [ 1814.841426] Lustre: server umount lustre-MDT0000 complete [ 1816.549876] LustreError: 51577:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1816.562113] LustreError: 51577:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 1818.664671] LustreError: 50429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754411 with bad export cookie 12506130454992375823 [ 1818.666186] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.672333] LustreError: 50429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1819.010563] Lustre: server umount lustre-MDT0001 complete [ 1830.345569] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1833.820274] Key type lgssc unregistered [ 1834.271959] LNet: 55395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.281580] LNetError: 55395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.315459] LNet: Removed LNI 192.168.202.156@tcp [ 1835.298434] Key type .llcrypt unregistered [ 1835.301174] Key type ._llcrypt unregistered [ 1876.383088] Key type ._llcrypt registered [ 1876.384466] Key type .llcrypt registered [ 1876.496904] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1878.000184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1878.119236] alg: No test for adler32 (adler32-zlib) [ 1879.264041] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 1879.644730] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1881.386991] Key type lgssc registered [ 1882.483413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1889.603530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1889.620516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.906813] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1890.944710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1891.159043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.752739] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1904.198805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.305372] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1904.396507] Lustre: 57601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1904.815760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.887541] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1918.401841] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1923.963919] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1931.433423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1931.670689] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1931.928778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.854161] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1939.651285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1946.833216] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1953.768287] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-07 0h15m44s [ 1955.767076] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1963.240861] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1965.887950] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1965.964124] LustreError: 57607:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1986.302886] Lustre: lustre-MDT0000: Client 1d83dc6f-5831-4caa-8374-b09f826f81e2 (at 192.168.202.56@tcp) reconnecting [ 2005.977440] LustreError: 57607:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2008.380386] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2011.203571] Lustre: DEBUG MARKER: done [ 2021.863151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2021.864969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.869347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2021.873927] LustreError: Skipped 1 previous similar message [ 2021.922021] Lustre: Skipped 1 previous similar message [ 2026.984308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2027.010115] Lustre: Skipped 2 previous similar messages [ 2031.072349] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2031.353741] Lustre: server umount lustre-OST0000 complete [ 2037.218418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.232598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.236494] Lustre: Skipped 1 previous similar message [ 2043.061757] Lustre: server umount lustre-MDT0000 complete [ 2047.768433] LustreError: 57592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754640 with bad export cookie 9893955958888807129 [ 2047.775326] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.781073] LustreError: 57592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2048.137634] Lustre: server umount lustre-MDT0001 complete [ 2062.129611] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2065.404298] Key type lgssc unregistered [ 2065.943598] LNet: 61937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.954768] LNetError: 61937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.012517] LNet: Removed LNI 192.168.202.156@tcp [ 2067.356066] Key type .llcrypt unregistered [ 2067.358989] Key type ._llcrypt unregistered [ 2122.312588] Key type ._llcrypt registered [ 2122.320294] Key type .llcrypt registered [ 2122.441459] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2123.928394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2124.015484] alg: No test for adler32 (adler32-zlib) [ 2125.300917] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 2125.587297] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2127.353352] Key type lgssc registered [ 2128.826912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.408044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.451403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.776607] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2133.796186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2134.152196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.276956] Lustre: Failing over lustre-MDT0000 [ 2135.793351] Lustre: server umount lustre-MDT0000 complete [ 2143.854888] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:18:53 (1788754733) [ 2152.626413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.032124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.886319] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2168.162724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.294488] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2168.356981] Lustre: 64746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2168.646857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2173.742955] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2182.733825] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2188.921625] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2195.973188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2196.203777] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2196.514560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.768743] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2204.311956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2212.513303] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2219.599919] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2230.242761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2230.262762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.266583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2230.286831] Lustre: Skipped 2 previous similar messages [ 2230.313491] Lustre: Skipped 2 previous similar messages [ 2233.935276] Lustre: server umount lustre-MDT0000 complete [ 2239.058592] LustreError: 64738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754831 with bad export cookie 10007834501230765049 [ 2239.088131] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.628081] Lustre: server umount lustre-MDT0001 complete [ 2243.601491] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2249.087611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2256.740652] Lustre: 63669:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754833/real 1788754833] req@00000000a83f62fb x1875645267223552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788754849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2256.769692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.877031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.960125] Lustre: 63671:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754837/real 1788754837] req@0000000088003c7a x1875645267223680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788754853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2261.984113] Lustre: 63672:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754838/real 1788754838] req@00000000edeab483 x1875645267223808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788754854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2264.243152] LustreError: 66593:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.249702] LustreError: 66593:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2264.349134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.472374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2265.319698] Lustre: 63669:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754841/real 1788754841] req@00000000013f483d x1875645267224064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788754857 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2265.377137] LustreError: 66594:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.405462] LustreError: 66593:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.421368] LustreError: 66593:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2269.486153] Lustre: 63670:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754846/real 1788754846] req@00000000cb390722 x1875645267224192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788754862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2269.498146] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2271.717637] LustreError: 68515:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.739038] LustreError: 68515:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2276.834394] LustreError: 68517:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.891767] LustreError: 68517:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2279.285419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.260553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.428502] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2286.877707] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2297.765250] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2304.481093] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2316.258679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.263130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2316.267612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.278127] Lustre: Skipped 1 previous similar message [ 2319.087533] Lustre: server umount lustre-MDT0000 complete [ 2321.392021] LustreError: 68653:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.414811] LustreError: 68653:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2323.533027] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754916 with bad export cookie 10007834501230767324 [ 2323.537231] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.543893] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2323.863254] Lustre: server umount lustre-MDT0001 complete [ 2331.932151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2342.880176] Lustre: 63672:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788754919/real 1788754919] req@00000000ab04756a x1875645267253376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788754935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2342.901901] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.916643] Lustre: Skipped 1 previous similar message [ 2343.249783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.089185] LustreError: 63668:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a16fac44 x1875645267254400/t0(0) o250->MGC192.168.202.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2349.486042] LustreError: 66593:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.585800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.656164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2354.809644] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2365.344280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.026912] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2371.061224] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2371.068827] Lustre: Skipped 1 previous similar message [ 2372.084807] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2378.766771] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2386.157334] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2396.647645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2396.668783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.697289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2396.712713] Lustre: Skipped 2 previous similar messages [ 2401.091549] Lustre: server umount lustre-MDT0000 complete [ 2402.786533] LustreError: 71201:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.838510] LustreError: 71201:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 9 previous similar messages [ 2407.860699] LustreError: 67659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755000 with bad export cookie 10007834501230768703 [ 2407.867814] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.894879] LustreError: 67659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2407.959681] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2407.983687] Lustre: Skipped 2 previous similar messages [ 2408.001842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2408.012692] Lustre: Skipped 3 previous similar messages [ 2408.796966] Lustre: server umount lustre-MDT0001 complete [ 2417.255384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2421.621785] Lustre: DEBUG MARKER: files compared the same [ 2424.321397] Lustre: server umount lustre-OST0000 complete [ 2444.860230] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2448.467906] Key type lgssc unregistered [ 2448.794928] LNet: 74480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2448.803600] LNetError: 74480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2448.825864] LNet: Removed LNI 192.168.202.156@tcp [ 2449.895921] Key type .llcrypt unregistered [ 2449.898788] Key type ._llcrypt unregistered [ 2477.764607] Key type ._llcrypt registered [ 2477.773933] Key type .llcrypt registered [ 2477.958286] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2479.311120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.428058] alg: No test for adler32 (adler32-zlib) [ 2480.777754] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 2481.121923] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2482.888425] Key type lgssc registered [ 2484.334733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2494.169219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2494.201194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.103779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.119780] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 2506.221381] LustreError: 75907:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.251855] LustreError: 75907:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2511.330634] LustreError: 75908:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.119656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.692134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2518.132498] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 2526.813979] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2533.065681] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2540.323294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2540.462533] LustreError: 77727:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.765818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.166624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2549.116800] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 2559.203481] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2571.745354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2571.751088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.764701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2572.770221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2572.770314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.300317] Lustre: server umount lustre-OST0000 complete [ 2583.008821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.025334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.038379] Lustre: Skipped 2 previous similar messages [ 2588.130108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.138654] Lustre: Skipped 1 previous similar message [ 2593.249585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.253307] Lustre: Skipped 1 previous similar message [ 2595.297534] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2595.476789] Lustre: server umount lustre-MDT0000 complete [ 2598.382453] LustreError: 76877:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.415331] LustreError: 76877:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2600.124788] LustreError: 75888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755192 with bad export cookie 12052757628730540357 [ 2600.129975] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.147919] LustreError: 75888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2600.532457] Lustre: server umount lustre-MDT0001 complete [ 2614.698655] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2618.327505] Key type lgssc unregistered [ 2618.665863] LNet: 79441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2618.678282] LNetError: 79441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2618.698325] LNet: Removed LNI 192.168.202.156@tcp [ 2619.979981] Key type .llcrypt unregistered [ 2619.981702] Key type ._llcrypt unregistered [ 2648.022250] Key type ._llcrypt registered [ 2648.024831] Key type .llcrypt registered [ 2648.120429] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2649.366649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2649.465025] alg: No test for adler32 (adler32-zlib) [ 2650.630320] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 2651.045946] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2652.840260] Key type lgssc registered [ 2654.302616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2663.082786] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2663.107466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.688541] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2683.809397] LustreError: 80829:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 4: rc = -110 [ 2710.560782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.359589] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2726.097501] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2736.040163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.223090] Lustre: *** cfs_fail_loc=706, val=0*** [ 2736.737915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2736.852602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2741.732393] LustreError: 80855:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.767254] LustreError: 80855:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2742.071184] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2745.831985] LustreError: 80854:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2745.861384] LustreError: 80854:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2750.975068] LustreError: 80855:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.007575] LustreError: 80855:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2752.327745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.933189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2759.510700] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2769.526738] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2775.578388] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2778.595561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.598325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2778.627862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2778.639612] Lustre: Skipped 1 previous similar message [ 2783.715787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2783.730098] Lustre: Skipped 1 previous similar message [ 2784.037472] Lustre: server umount lustre-OST0000 complete [ 2788.833933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2788.844823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.852212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2788.858609] Lustre: Skipped 2 previous similar messages [ 2788.878625] Lustre: Skipped 1 previous similar message [ 2793.954155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2793.969688] Lustre: Skipped 1 previous similar message [ 2799.078780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.084254] Lustre: Skipped 1 previous similar message [ 2803.168161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2803.352721] Lustre: server umount lustre-MDT0000 complete [ 2804.205561] LustreError: 81921:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.253547] LustreError: 81921:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2807.006848] LustreError: 80837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755399 with bad export cookie 11866173857179647130 [ 2807.014947] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2807.033959] LustreError: 80837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2807.735666] Lustre: server umount lustre-MDT0001 complete [ 2821.842291] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2826.265949] Key type lgssc unregistered [ 2826.575445] LNet: 84446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2826.580348] LNetError: 84446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2826.607734] LNet: Removed LNI 192.168.202.156@tcp [ 2827.711371] Key type .llcrypt unregistered [ 2827.713529] Key type ._llcrypt unregistered [ 2855.869367] Key type ._llcrypt registered [ 2855.871721] Key type .llcrypt registered [ 2856.020097] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2857.105077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2857.251186] alg: No test for adler32 (adler32-zlib) [ 2858.313354] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 2858.626727] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2860.288231] Key type lgssc registered [ 2862.067627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2873.216849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2873.246782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.569990] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2889.146781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.846234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2893.037330] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2895.333795] LustreError: 86548:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.344113] LustreError: 86548:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2899.430047] LustreError: 86552:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.463572] LustreError: 86552:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2900.799850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2901.119366] LustreError: 86548:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.130013] LustreError: 86548:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2901.138939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.594824] LustreError: 86547:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.617359] LustreError: 86547:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2908.795902] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2911.719561] LustreError: 86551:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.748932] LustreError: 86551:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2918.068718] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2918.096781] Lustre: MGS: Client 205a616c-f4ff-4371-86eb-905a65cadd14 (at 0@lo) reconnecting [ 2918.114325] LustreError: 86562:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2918.414132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.525068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2924.081742] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2940.906163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2940.916249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.933570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.972119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2943.979430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.994088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.029120] Lustre: Skipped 1 previous similar message [ 2946.754307] Lustre: server umount lustre-OST0000 complete [ 2954.209615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2954.212061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.219253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2954.252059] Lustre: Skipped 1 previous similar message [ 2957.560258] Lustre: server umount lustre-MDT0000 complete [ 2959.336576] LustreError: 86547:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.347678] LustreError: 86547:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2961.930738] Lustre: server umount lustre-MDT0001 complete [ 2965.837491] Lustre: server umount MGS complete [ 2977.933564] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:32:48 (1788755568) [ 3012.749621] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3024.498190] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3066.530265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.126466] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3083.387931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.723675] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3083.794462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3083.984515] Lustre: lustre-MDT0001: new disk, initializing [ 3084.144265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3084.195543] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3084.208384] Lustre: 93740:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3084.227417] LustreError: 93740:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3090.250871] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3101.441391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.768190] Lustre: lustre-OST0000: new disk, initializing [ 3101.772163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3101.779911] Lustre: 94507:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3101.889048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3109.177775] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3111.521532] Lustre: 94920:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006a7b5517 x1875646035829632/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788755720 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3119.910408] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3127.776909] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3127.806548] Lustre: MGS: Client 4aa0ce74-99f8-4a76-b656-c80b9f5450bf (at 0@lo) reconnecting [ 3127.839458] LustreError: 93726:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3127.919672] Lustre: 93009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3128.007274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3128.185256] Lustre: lustre-MDT0000: new disk, initializing [ 3128.537646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3128.584778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3128.608581] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3129.634090] Lustre: 95576:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000098c83400 x1875646035836800/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788755738 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3134.817970] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3135.265029] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3138.798110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3138.813533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3138.907889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3164.643543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3164.661631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3164.686715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3164.695498] Lustre: Skipped 2 previous similar messages [ 3169.453747] Lustre: server umount lustre-OST0000 complete [ 3177.954278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3177.955704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.959748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3177.959758] Lustre: Skipped 2 previous similar messages [ 3177.966339] LustreError: Skipped 1 previous similar message [ 3178.007120] Lustre: Skipped 2 previous similar messages [ 3181.189716] Lustre: server umount lustre-MDT0000 complete [ 3183.078204] LustreError: 93711:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.105313] LustreError: 93711:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3192.870348] Lustre: server umount lustre-MDT0001 complete [ 3197.879765] Lustre: server umount MGS complete [ 3214.545441] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:36:44 (1788755804) [ 3238.437286] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3241.716427] Key type lgssc unregistered [ 3242.144605] LNet: 98090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.161322] LNetError: 98090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.180819] LNet: Removed LNI 192.168.202.156@tcp [ 3243.311786] Key type .llcrypt unregistered [ 3243.315146] Key type ._llcrypt unregistered [ 3264.310325] Key type ._llcrypt registered [ 3264.312728] Key type .llcrypt registered [ 3264.435847] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3265.906750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3266.112208] alg: No test for adler32 (adler32-zlib) [ 3267.406191] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 3267.746711] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3269.472232] Key type lgssc registered [ 3271.211324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3280.770989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3280.788150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.141748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.105286] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3291.620579] LustreError: 99310:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.632984] LustreError: 99310:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3296.739027] LustreError: 99311:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.782269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.466835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3303.534133] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3310.887945] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3319.189608] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3326.653591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.209798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.676804] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3344.434447] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3351.811889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3367.398155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3367.409515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3367.437482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3371.123701] Lustre: server umount lustre-OST0000 complete [ 3377.639581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.661278] Lustre: Skipped 1 previous similar message [ 3377.671955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.684060] Lustre: Skipped 2 previous similar messages [ 3382.482721] Lustre: server umount lustre-MDT0000 complete [ 3382.768126] LustreError: 99307:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.794020] LustreError: 99307:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3386.671657] LustreError: 99290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755979 with bad export cookie 5025184244164193527 [ 3386.675424] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3386.681949] LustreError: 99290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3387.319593] Lustre: server umount lustre-MDT0001 complete [ 3402.133275] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3405.330977] Key type lgssc unregistered [ 3405.655039] LNet: 102869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3405.663510] LNetError: 102869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3405.682300] LNet: Removed LNI 192.168.202.156@tcp [ 3406.866452] Key type .llcrypt unregistered [ 3406.872427] Key type ._llcrypt unregistered [ 3425.808617] Key type ._llcrypt registered [ 3425.810593] Key type .llcrypt registered [ 3425.893207] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3427.167534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3427.228690] alg: No test for adler32 (adler32-zlib) [ 3428.548274] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 3428.838746] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3430.488534] Key type lgssc registered [ 3432.000890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3439.028427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3439.031405] LustreError: 103966:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3439.036827] Lustre: Skipped 1 previous similar message [ 3439.036858] LustreError: 103967:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3439.064124] LustreError: 103966:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3439.072224] LustreError: 103966:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3439.075534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.083215] LustreError: 103966:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3440.719039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3449.353406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.594918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3455.101176] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3461.710471] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing lsmod [ 3465.767235] LustreError: 105162:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3465.814476] LustreError: 105175:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3465.828339] LustreError: 105162:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4939 [ 3465.839171] LustreError: 105162:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3465.851515] LustreError: 105162:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3465.866839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3466.274527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.305870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3470.321521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.346656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.330455] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.333853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.357167] Lustre: Skipped 1 previous similar message [ 3471.374077] Lustre: Skipped 2 previous similar messages [ 3474.468682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.481149] Lustre: Skipped 2 previous similar messages [ 3475.887451] Lustre: server umount lustre-MDT0000 complete [ 3479.530262] LustreError: 104005:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.565311] LustreError: 104005:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 3480.300488] LustreError: 105213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756073 with bad export cookie 6286316642575165462 [ 3480.311963] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3480.313986] LustreError: 105213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3480.696816] LustreError: 104435:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a47bfcaf x1875646633497728/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3480.773758] Lustre: server umount lustre-MDT0001 complete [ 3495.134741] Lustre: server umount lustre-OST0000 complete [ 3507.285069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.338435] LustreError: 106240:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3508.442263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.842806] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3513.831714] LustreError: 106241:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.948923] LustreError: 106240:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3524.065751] LustreError: 106241:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3524.239511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.760663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3531.089102] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3541.138673] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3547.745756] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3555.425183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.168311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3563.294554] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3571.894757] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3578.726209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3589.090404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3589.093813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.095816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.126021] Lustre: Skipped 1 previous similar message [ 3594.217478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.224720] Lustre: Skipped 2 previous similar messages [ 3595.137568] Lustre: server umount lustre-OST0000 complete [ 3601.388879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3601.406058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.451378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.472058] Lustre: Skipped 2 previous similar messages [ 3606.133817] Lustre: server umount lustre-MDT0000 complete [ 3609.575668] LustreError: 106237:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.609915] LustreError: 106237:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3611.317327] LustreError: 108082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756204 with bad export cookie 6286316642575166162 [ 3611.318555] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3611.323634] LustreError: 108082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3611.638611] Lustre: server umount lustre-MDT0001 complete [ 3622.310770] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3627.329286] Key type lgssc unregistered [ 3627.745894] LNet: 109797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3627.758944] LNetError: 109797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3628.775876] LNet: Removed LNI 192.168.202.156@tcp [ 3629.668289] Key type .llcrypt unregistered [ 3629.670559] Key type ._llcrypt unregistered [ 3662.131264] Key type ._llcrypt registered [ 3662.135427] Key type .llcrypt registered [ 3662.334926] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3663.377661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3663.391502] alg: No test for adler32 (adler32-zlib) [ 3664.886112] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 3665.339664] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3667.200381] Key type lgssc registered [ 3668.866882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3679.689908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3679.714702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.330782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3687.608354] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3691.495871] LustreError: 111208:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.533838] LustreError: 111208:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3696.611532] LustreError: 111209:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3698.005279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.436305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3704.190253] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3715.396309] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3721.643410] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3730.697088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3731.212328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.988195] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3744.550838] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3752.712782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3761.456338] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3765.436784] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3773.411358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.419077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3773.423625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3773.436857] Lustre: Skipped 1 previous similar message [ 3778.532786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3778.537457] Lustre: Skipped 1 previous similar message [ 3783.649913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3783.663149] Lustre: Skipped 2 previous similar messages [ 3784.161464] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3784.339578] Lustre: server umount lustre-OST0000 complete [ 3790.823651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3790.834658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.847114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.898329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.009668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.021611] Lustre: Skipped 3 previous similar messages [ 3804.133615] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3804.346391] Lustre: server umount lustre-MDT0000 complete [ 3809.252428] LustreError: 111208:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3809.277675] LustreError: 111208:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3809.442543] LustreError: 111189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756402 with bad export cookie 13080710869886234632 [ 3809.444112] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3809.450066] LustreError: 111189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3809.874345] Lustre: server umount lustre-MDT0001 complete [ 3821.276573] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3824.789530] Key type lgssc unregistered [ 3825.141809] LNet: 114879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3825.149949] LNetError: 114879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3826.218776] LNet: Removed LNI 192.168.202.156@tcp [ 3827.622884] Key type .llcrypt unregistered [ 3827.625805] Key type ._llcrypt unregistered [ 3852.017074] Key type ._llcrypt registered [ 3852.022101] Key type .llcrypt registered [ 3852.235752] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3853.823860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3853.988703] alg: No test for adler32 (adler32-zlib) [ 3855.291799] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 3855.581705] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3857.408241] Key type lgssc registered [ 3858.982980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3869.431167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3869.454259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.039435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3876.827877] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3880.491787] LustreError: 116099:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.505849] LustreError: 116099:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3885.540931] LustreError: 116100:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.440846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3888.298553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3892.726949] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3902.490405] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3909.595852] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3917.909150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3918.309291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.141392] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3932.172305] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3940.870749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3952.106037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3952.113679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.136596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3953.127518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3953.135752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.148579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3956.199447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3956.211180] Lustre: Skipped 1 previous similar message [ 3957.800839] Lustre: server umount lustre-OST0000 complete [ 3964.899327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3964.911430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.931236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3968.068481] Lustre: server umount lustre-MDT0000 complete [ 3971.553973] LustreError: 118884:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.580539] LustreError: 118884:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3972.314816] LustreError: 116081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756565 with bad export cookie 11419956404055434171 [ 3972.331218] LustreError: 116081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3972.334259] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3972.837562] Lustre: server umount lustre-MDT0001 complete [ 3986.802590] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3989.804941] Key type lgssc unregistered [ 3990.206362] LNet: 119656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3990.211369] LNetError: 119656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3990.240359] LNet: Removed LNI 192.168.202.156@tcp [ 3991.465850] Key type .llcrypt unregistered [ 3991.468740] Key type ._llcrypt unregistered [ 4020.171256] Key type ._llcrypt registered [ 4020.174736] Key type .llcrypt registered [ 4020.280706] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4021.409776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4021.486069] alg: No test for adler32 (adler32-zlib) [ 4022.549489] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 4022.847900] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4024.563902] Key type lgssc registered [ 4026.192218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4035.006572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4035.031302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.656856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.047508] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4046.819107] LustreError: 121071:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.836317] LustreError: 121071:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4051.441613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.907431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4056.954492] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4065.890692] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4071.972472] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4079.772392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.221263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.325790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4087.681966] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4096.360294] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4104.065077] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4108.554790] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4108.565798] Lustre: Skipped 2 previous similar messages [ 4109.923686] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4109.960560] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4113.599055] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4113.606478] Lustre: Skipped 2 previous similar messages [ 4119.652480] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4119.663027] Lustre: Skipped 1 previous similar message [ 4126.795332] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4126.807460] Lustre: Skipped 2 previous similar messages [ 4129.669670] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp [ 4142.392276] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.56/24@tcp in log lustre-MDT0000 [ 4142.404938] Lustre: Skipped 2 previous similar messages [ 4150.881159] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.56/24@tcp [ 4150.896810] Lustre: Skipped 1 previous similar message [ 4158.440863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4158.452605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4158.453967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4163.555840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4163.570061] Lustre: Skipped 2 previous similar messages [ 4164.773059] Lustre: server umount lustre-OST0000 complete [ 4169.698333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4169.713224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.747419] Lustre: Skipped 1 previous similar message [ 4169.762923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4173.793651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.798515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4175.564590] Lustre: server umount lustre-MDT0000 complete [ 4178.914814] LustreError: 121067:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.943372] LustreError: 121067:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4180.299381] LustreError: 121052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756773 with bad export cookie 4191875067327634457 [ 4180.307622] LustreError: 121052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4180.308187] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4180.846948] Lustre: server umount lustre-MDT0001 complete [ 4194.944604] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 4198.245771] Key type lgssc unregistered [ 4198.659868] LNet: 126021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4198.684368] LNetError: 126021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4198.727840] LNet: Removed LNI 192.168.202.156@tcp [ 4199.929397] Key type .llcrypt unregistered [ 4199.935292] Key type ._llcrypt unregistered [ 4232.886477] Key type ._llcrypt registered [ 4232.889083] Key type .llcrypt registered [ 4233.008171] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4234.276345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4234.441562] alg: No test for adler32 (adler32-zlib) [ 4235.543418] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 4235.817204] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4237.640263] Key type lgssc registered [ 4239.448933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4248.066673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4248.105398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4249.757254] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4249.796368] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.56/24@tcp [ 4249.846202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.947934] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4259.301257] LustreError: 127580:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.317709] LustreError: 127580:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4264.420162] LustreError: 127579:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.037732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4266.380382] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4266.469510] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.56/24@tcp [ 4266.587096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4272.141292] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4281.008608] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4287.618941] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4295.829633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4296.266479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.183745] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4311.838678] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4319.824939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4335.081202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4335.092803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.102261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4335.116403] Lustre: Skipped 1 previous similar message [ 4335.131274] Lustre: Skipped 1 previous similar message [ 4339.643702] Lustre: server umount lustre-OST0000 complete [ 4345.319382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.340446] Lustre: Skipped 1 previous similar message [ 4345.342393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4345.376168] Lustre: Skipped 1 previous similar message [ 4350.436458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4350.449665] Lustre: Skipped 1 previous similar message [ 4354.530384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.539183] Lustre: Skipped 1 previous similar message [ 4358.112134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4358.300819] Lustre: server umount lustre-MDT0000 complete [ 4359.651071] LustreError: 127574:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.683245] LustreError: 127574:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4362.140956] LustreError: 127559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756954 with bad export cookie 10596530680935707076 [ 4362.151624] LustreError: 127559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4362.152260] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4362.436359] Lustre: server umount lustre-MDT0001 complete [ 4373.003365] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 4375.897950] Key type lgssc unregistered [ 4376.300059] LNet: 131189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4376.308044] LNetError: 131189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4376.321534] LNet: Removed LNI 192.168.202.156@tcp [ 4377.231906] Key type .llcrypt unregistered [ 4377.235953] Key type ._llcrypt unregistered [ 4407.226318] Key type ._llcrypt registered [ 4407.230662] Key type .llcrypt registered [ 4407.383949] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4421.531814] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4422.563931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4422.853355] alg: No test for adler32 (adler32-zlib) [ 4423.871831] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 4424.091871] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4425.873250] Key type lgssc registered [ 4427.140804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4463.905396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4463.940783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.429927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4465.483248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4465.634446] Lustre: lustre-MDT0000: new disk, initializing [ 4465.771644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.805323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4471.023787] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4482.330138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.489895] Lustre: 134590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4482.541865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4482.545243] Lustre: Skipped 1 previous similar message [ 4482.596442] Lustre: lustre-MDT0001: new disk, initializing [ 4482.682842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4482.721287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4482.743346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4488.233219] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4498.598487] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4504.312936] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4511.493902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4511.809826] Lustre: lustre-OST0000: new disk, initializing [ 4511.815631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4511.825383] Lustre: 136561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4511.916765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4519.001394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4519.016473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4519.080483] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4519.238578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4530.058845] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4535.184512] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4535.357464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4541.037974] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4541.255229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4545.007681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4545.017689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.030844] Lustre: Skipped 1 previous similar message [ 4545.036605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4545.043263] Lustre: Skipped 1 previous similar message [ 4548.069080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4548.079606] Lustre: Skipped 1 previous similar message [ 4549.784269] Lustre: server umount lustre-OST0000 complete [ 4558.306964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4558.310709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4558.326990] Lustre: Skipped 1 previous similar message [ 4558.339903] Lustre: Skipped 1 previous similar message [ 4560.913580] Lustre: server umount lustre-MDT0000 complete [ 4564.460153] LustreError: 134582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757157 with bad export cookie 8106557067199248648 [ 4564.461433] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4564.473421] LustreError: 134582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4564.833531] Lustre: server umount lustre-MDT0001 complete [ 4573.489102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4574.341210] LustreError: 138778:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4574.384118] LustreError: 138778:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4574.532218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4579.815869] LustreError: 138779:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.620061] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4584.944501] LustreError: 138778:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.049929] LustreError: 138779:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.140180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4590.654164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4594.994733] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4603.569976] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4609.301529] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4616.263383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4616.715857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4623.518941] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4631.594045] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4636.853548] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4637.131270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4642.434699] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4642.822252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4651.862922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4652.131213] Lustre: lustre-OST0001: new disk, initializing [ 4652.138825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4652.153316] Lustre: 142152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4652.259445] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4659.469803] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4662.394365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4662.403624] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4662.524200] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4668.459442] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4668.779229] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4674.287622] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4674.494183] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4679.840754] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90facb13b000.ost_server_uuid 50 [ 4681.418423] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90facb13b000.ost_server_uuid in IDLE state after 0 sec [ 4691.445601] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.452183] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4691.460979] Lustre: Skipped 1 previous similar message [ 4691.485617] Lustre: Skipped 1 previous similar message [ 4696.549500] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4697.263849] Lustre: server umount lustre-OST0001 complete [ 4701.666157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4701.667612] LustreError: 140600:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4701.679387] LustreError: Skipped 1 previous similar message [ 4701.705839] LustreError: 140600:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4701.708797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4706.791772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4706.796546] Lustre: Skipped 4 previous similar messages [ 4707.598618] Lustre: server umount lustre-OST0000 complete [ 4711.906517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.930432] Lustre: Skipped 2 previous similar messages [ 4717.777801] Lustre: server umount lustre-MDT0000 complete [ 4722.023290] LustreError: 138760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757314 with bad export cookie 8106557067199249530 [ 4722.036784] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4722.045221] LustreError: 138760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4722.400165] Lustre: server umount lustre-MDT0001 complete [ 4742.120150] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 4745.426782] Key type lgssc unregistered [ 4745.922614] LNet: 145168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4745.928693] LNetError: 145168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4745.954095] LNet: Removed LNI 192.168.202.156@tcp [ 4746.943478] Key type .llcrypt unregistered [ 4746.946632] Key type ._llcrypt unregistered [ 4788.947964] Key type ._llcrypt registered [ 4788.950533] Key type .llcrypt registered [ 4789.095399] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4807.473321] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4808.918186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4808.977137] alg: No test for adler32 (adler32-zlib) [ 4810.658751] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 4811.135579] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4812.865364] Key type lgssc registered [ 4814.789802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4855.547451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4855.592588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4857.039833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4857.067853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4857.144802] Lustre: lustre-MDT0000: new disk, initializing [ 4857.248996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.270597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4862.822166] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4873.127240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4873.341183] Lustre: 149320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4873.436615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4873.439190] Lustre: Skipped 1 previous similar message [ 4873.564892] Lustre: lustre-MDT0001: new disk, initializing [ 4873.663647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4873.738127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4873.762896] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4878.800196] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4888.666469] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4895.331430] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4903.561955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.078383] Lustre: lustre-OST0000: new disk, initializing [ 4904.081187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4904.093206] Lustre: 151287:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4904.220435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4910.579809] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4912.274732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4912.297098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4912.446850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4920.943340] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4926.309962] Lustre: Failing over lustre-OST0000 [ 4927.972540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4927.976642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.979315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4927.982079] Lustre: Skipped 1 previous similar message [ 4928.015240] Lustre: Skipped 1 previous similar message [ 4928.638379] Lustre: server umount lustre-OST0000 complete [ 4948.737399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4949.239229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4949.281549] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4955.830187] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4955.863668] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4960.136776] Lustre: Failing over lustre-MDT0000 [ 4960.464743] Lustre: server umount lustre-MDT0000 complete [ 4960.745474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4960.749674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.788865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.801405] Lustre: Skipped 1 previous similar message [ 4966.051657] LustreError: 152120:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4966.067612] LustreError: 152120:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 4967.906189] LustreError: 149325:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.906189] LustreError: 149331:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.971799] LustreError: 149325:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4971.167386] LustreError: 149327:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.287619] LustreError: 149325:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.310342] LustreError: 149325:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4979.169390] Lustre: 147541:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788757555/real 1788757555] req@000000001e2ee6f4 x1875648082654848/t0(0) o400->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788757571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4979.200157] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4979.765135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4981.409801] LustreError: 149326:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.435222] LustreError: 149326:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 4988.545103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4988.580875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.631233] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4991.646453] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4993.663298] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4994.083180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4994.118216] Lustre: lustre-OST0000: Recovery over after 0:39, of 3 clients 3 recovered and 0 were evicted. [ 4994.196846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5004.277446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5004.292590] Lustre: Skipped 1 previous similar message [ 5004.295473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5004.313847] Lustre: Skipped 1 previous similar message [ 5008.759206] Lustre: server umount lustre-OST0000 complete [ 5014.502402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.516471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5014.517778] Lustre: Skipped 1 previous similar message [ 5019.618834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5019.627800] Lustre: Skipped 2 previous similar messages [ 5024.739859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.748114] Lustre: Skipped 1 previous similar message [ 5028.320160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5028.519302] Lustre: server umount lustre-MDT0000 complete [ 5029.862906] LustreError: 153793:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5029.881321] LustreError: 153793:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 8 previous similar messages [ 5032.571724] LustreError: 149311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788757625 with bad export cookie 10036760032339738152 [ 5032.583577] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5032.584992] LustreError: 149311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5032.949399] Lustre: server umount lustre-MDT0001 complete [ 5048.130172] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 5051.504157] Key type lgssc unregistered [ 5051.801356] LNet: 154946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5051.808839] LNetError: 154946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5051.831514] LNet: Removed LNI 192.168.202.156@tcp [ 5052.494074] Key type .llcrypt unregistered [ 5052.496709] Key type ._llcrypt unregistered [ 5093.615762] Key type ._llcrypt registered [ 5093.618807] Key type .llcrypt registered [ 5093.725149] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5095.317179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5095.349351] alg: No test for adler32 (adler32-zlib) [ 5096.654566] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 5097.063644] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 5098.832213] Key type lgssc registered [ 5100.297466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5109.126687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5109.151546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5110.801395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.888491] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5121.002815] LustreError: 156461:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.033527] LustreError: 156461:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5125.807555] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5126.118816] LustreError: 156462:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5129.861719] LustreError: 156490:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5129.873406] LustreError: 156490:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5129.896386] LustreError: 156490:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 5130.222853] Lustre: server umount lustre-MDT0000 complete [ 5145.875962] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5155.917405] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5191.551040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.838175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5191.873647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5191.978390] Lustre: lustre-MDT0000: new disk, initializing [ 5192.050913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5192.068349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5196.812455] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5208.165889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.390944] Lustre: 159795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5208.458917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5208.461822] Lustre: Skipped 1 previous similar message [ 5208.519962] Lustre: lustre-MDT0001: new disk, initializing [ 5208.608776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5208.641929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5208.667727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5214.444755] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5224.468788] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5230.521967] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5237.830617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5238.167466] Lustre: lustre-OST0000: new disk, initializing [ 5238.171301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5238.185989] Lustre: 161766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5238.305120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5244.310610] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5244.570135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5244.576221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5244.717651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5255.736864] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5261.423631] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5263.969697] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:10:54 (1788757854) [ 5505.506452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5505.518540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5505.551528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5506.019580] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.045963] Lustre: Skipped 1 previous similar message [ 5506.067651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5506.080756] Lustre: Skipped 2 previous similar messages [ 5511.138031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5516.258159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5516.269513] Lustre: Skipped 4 previous similar messages [ 5517.281245] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5517.742952] Lustre: server umount lustre-MDT0000 complete [ 5521.384850] LustreError: 160846:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.412439] LustreError: 160846:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5522.585366] LustreError: 159789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758115 with bad export cookie 4880369769507720727 [ 5522.592894] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5522.603185] LustreError: 159789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5526.498608] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5526.501737] LustreError: 159805:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5526.532491] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5529.229621] Lustre: server umount lustre-MDT0001 complete [ 5540.266976] Lustre: server umount lustre-OST0000 complete [ 5552.138308] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5564.700710] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5607.012373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5607.301744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5607.372774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5607.457644] Lustre: lustre-MDT0000: new disk, initializing [ 5607.539211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.574471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5613.704594] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5625.297138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5625.465297] Lustre: 166429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5625.525741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5625.529490] Lustre: Skipped 1 previous similar message [ 5625.639090] Lustre: lustre-MDT0001: new disk, initializing [ 5625.764071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5625.821330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5625.833642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5632.137867] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5642.548504] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5649.894281] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5657.259899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5657.682724] Lustre: lustre-OST0000: new disk, initializing [ 5657.689669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5657.705107] Lustre: 168398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5657.838556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.752984] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5665.419961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5665.438492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5665.559253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5674.970681] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5679.415460] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5679.610200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5684.604451] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5684.925420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5691.366478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.367462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5691.397407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5691.397414] Lustre: Skipped 1 previous similar message [ 5693.676280] Lustre: server umount lustre-OST0000 complete [ 5701.602191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.611521] Lustre: Skipped 1 previous similar message [ 5711.848492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.856353] Lustre: Skipped 6 previous similar messages [ 5712.352358] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5712.642367] Lustre: server umount lustre-MDT0000 complete [ 5716.968611] LustreError: 166440:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5716.977433] LustreError: 166440:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5717.898102] LustreError: 167225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758310 with bad export cookie 4880369769507879557 [ 5717.906966] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5717.929718] LustreError: 167225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5718.624403] Lustre: server umount lustre-MDT0001 complete [ 5731.009092] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:18:40 (1788758320) [ 5751.241462] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5763.043837] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5802.670747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.882273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5802.909355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5802.968303] Lustre: lustre-MDT0000: new disk, initializing [ 5803.059782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.075386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5807.998674] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5820.374269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5820.586620] Lustre: 173157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5821.165582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5826.780235] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5837.172549] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5843.370335] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5851.003785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5851.403919] Lustre: lustre-OST0000: new disk, initializing [ 5851.406490] Lustre: Skipped 1 previous similar message [ 5851.413217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5851.416969] Lustre: Skipped 2 previous similar messages [ 5851.422071] Lustre: 175133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5857.951375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5857.964519] Lustre: Skipped 1 previous similar message [ 5857.970432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5858.167576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5858.911274] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5870.049645] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5878.754451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5878.765160] LustreError: Skipped 1 previous similar message [ 5878.768452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.782918] Lustre: Skipped 1 previous similar message [ 5878.788623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5878.792978] Lustre: Skipped 1 previous similar message [ 5884.314455] Lustre: server umount lustre-OST0000 complete [ 5903.328852] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5903.605784] Lustre: server umount lustre-MDT0000 complete [ 5904.368783] LustreError: 175951:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.384161] LustreError: 175951:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5907.582105] LustreError: 173149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758500 with bad export cookie 4880369769507880439 [ 5907.602832] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5907.607638] LustreError: 173149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5908.137245] Lustre: server umount lustre-MDT0001 complete [ 5917.531970] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:21:47 (1788758507) [ 5938.506726] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5952.636342] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5996.068351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.457549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5996.533433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5996.679852] Lustre: lustre-MDT0000: new disk, initializing [ 5996.815559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.825689] Lustre: Skipped 2 previous similar messages [ 5996.844686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6003.273556] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6016.227196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6016.335588] Lustre: 179460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6016.559758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6021.647802] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6032.547107] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6038.117691] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6045.676542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6046.060587] Lustre: 181439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6052.499675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6052.744275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6053.713168] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6065.226275] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6078.440622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.446037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.446292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.446297] LustreError: Skipped 1 previous similar message [ 6078.448069] Lustre: Skipped 4 previous similar messages [ 6078.464058] Lustre: Skipped 11 previous similar messages [ 6080.661359] Lustre: server umount lustre-OST0000 complete [ 6088.164125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6088.169674] LustreError: Skipped 1 previous similar message [ 6091.301867] Lustre: server umount lustre-MDT0000 complete [ 6093.793298] LustreError: 179472:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6093.807208] LustreError: 179472:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6095.515458] LustreError: 179454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758688 with bad export cookie 4880369769507881524 [ 6095.520163] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6095.524932] LustreError: 179454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6095.916801] Lustre: server umount lustre-MDT0001 complete [ 6107.611451] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6110.796224] Key type lgssc unregistered [ 6111.043872] LNet: 183295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6111.048926] LNetError: 183295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6111.060272] LNet: Removed LNI 192.168.202.156@tcp [ 6111.967103] Key type .llcrypt unregistered [ 6111.970345] Key type ._llcrypt unregistered [ 6141.360438] Key type ._llcrypt registered [ 6141.362461] Key type .llcrypt registered [ 6141.462331] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6142.565984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6142.712119] alg: No test for adler32 (adler32-zlib) [ 6143.751823] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 6143.952862] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6145.728797] Key type lgssc registered [ 6147.408903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6157.748128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6157.830178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.513626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6165.559316] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6170.085328] LustreError: 184710:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.112381] LustreError: 184710:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6175.215905] LustreError: 184711:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6175.681070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.159312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6180.858408] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6188.650215] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6195.224666] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6205.679965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6206.016521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6213.802452] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6222.869580] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6229.562858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6249.960353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6249.968940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.971596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6249.988952] Lustre: Skipped 1 previous similar message [ 6250.001632] Lustre: Skipped 1 previous similar message [ 6253.727535] Lustre: server umount lustre-OST0000 complete [ 6260.194168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.200569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.220576] Lustre: Skipped 1 previous similar message [ 6260.258457] Lustre: Skipped 1 previous similar message [ 6265.328905] LustreError: 185951:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.634557] Lustre: server umount lustre-MDT0000 complete [ 6269.412314] LustreError: 184710:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6269.432036] LustreError: 184710:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6270.506419] LustreError: 184690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758863 with bad export cookie 13330603865671074662 [ 6270.509236] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6270.521949] LustreError: 184690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6270.870684] Lustre: server umount lustre-MDT0001 complete [ 6281.981601] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6284.626116] Key type lgssc unregistered [ 6285.296698] LNet: 188268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6285.307658] LNetError: 188268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6285.330846] LNet: Removed LNI 192.168.202.156@tcp [ 6286.782121] Key type .llcrypt unregistered [ 6286.785418] Key type ._llcrypt unregistered [ 6315.495465] Key type ._llcrypt registered [ 6315.499683] Key type .llcrypt registered [ 6315.703115] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6317.667874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6317.907308] alg: No test for adler32 (adler32-zlib) [ 6319.065439] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 6319.398880] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6321.184630] Key type lgssc registered [ 6322.595601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6332.364760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6332.401763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.968878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.337433] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6344.166972] LustreError: 189683:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6344.212598] LustreError: 189683:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6349.291698] LustreError: 189682:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.558820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.006427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6355.522807] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6364.763725] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6371.235982] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6378.802194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6378.989440] LustreError: 191505:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6379.277376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.679048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6387.575370] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6395.490857] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.253119] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 6405.092574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6405.097698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6405.110060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6405.117175] LustreError: Skipped 1 previous similar message [ 6405.156112] Lustre: Skipped 1 previous similar message [ 6407.328220] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 6407.341230] Lustre: Skipped 1 previous similar message [ 6408.084087] Lustre: server umount lustre-OST0000 complete [ 6418.317596] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90fac7aa5000.ost_server_uuid 50 [ 6420.247223] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6436.840191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6436.849480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.890751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.907485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.402467] Lustre: server umount lustre-MDT0000 complete [ 6445.009575] LustreError: 189662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759037 with bad export cookie 9395459191453223157 [ 6445.017827] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6445.377428] Lustre: server umount lustre-MDT0001 complete [ 6455.401617] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:30:45 (1788759045) [ 6462.635822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.211281] LustreError: 193599:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.287834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6468.374707] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6468.578087] LustreError: 193600:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.657917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.794224] LustreError: 193599:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.821637] LustreError: 193599:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6478.118425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6483.534785] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6491.220619] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6497.202817] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6505.148648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.618394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.708951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6512.640936] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6520.194029] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6529.114911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6529.437697] Lustre: lustre-OST0001: new disk, initializing [ 6529.443446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6529.450987] Lustre: 196440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6529.575227] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6535.854706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6535.869466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6536.073793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6537.775342] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6548.619424] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6554.091907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6554.108587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6554.110102] LustreError: Skipped 1 previous similar message [ 6554.111482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6554.111488] Lustre: Skipped 2 previous similar messages [ 6554.175868] Lustre: Skipped 1 previous similar message [ 6557.861515] Lustre: server umount lustre-OST0000 complete [ 6559.924850] LustreError: 195420:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6565.647793] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6565.877888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6570.966599] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6571.236584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6587.368501] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6587.373477] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.380387] LustreError: Skipped 1 previous similar message [ 6587.385900] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6587.414718] Lustre: Skipped 1 previous similar message [ 6587.448754] Lustre: Skipped 5 previous similar messages [ 6590.822993] Lustre: server umount lustre-OST0001 complete [ 6596.067713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6596.072718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.942662] Lustre: server umount lustre-MDT0000 complete [ 6602.734179] LustreError: 193600:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6602.752799] LustreError: 193600:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 17 previous similar messages [ 6605.207798] LustreError: 193578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759197 with bad export cookie 9395459191453224228 [ 6605.216342] LustreError: 193578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6605.222298] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6605.516283] Lustre: server umount lustre-MDT0001 complete [ 6633.996079] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:33:44 (1788759224) [ 6641.174882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6641.660820] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6641.720412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6642.155329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6648.422045] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6659.929365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6660.045709] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6660.145257] Lustre: 199883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6666.113164] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6673.345941] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6680.199663] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6687.256845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6687.485508] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6694.080464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6694.731984] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6703.306698] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6711.122806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6711.199236] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6711.539449] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6711.543674] Lustre: Skipped 2 previous similar messages [ 6718.246023] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6719.689466] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6728.425617] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6733.800507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6733.816814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6733.825033] Lustre: Skipped 2 previous similar messages [ 6733.848437] Lustre: Skipped 4 previous similar messages [ 6737.972950] Lustre: server umount lustre-OST0000 complete [ 6738.931830] LustreError: 201732:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.948531] LustreError: 201732:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6758.718299] Lustre: server umount lustre-OST0001 complete [ 6769.637477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6769.640061] Lustre: Skipped 3 previous similar messages [ 6773.991913] Lustre: server umount lustre-MDT0000 complete [ 6775.648971] Lustre: 188991:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788759352/real 1788759352] req@00000000d9b05017 x1875649665079296/t0(0) o400->lustre-OST0001-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788759368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6775.691925] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.721528] Lustre: Skipped 2 previous similar messages [ 6776.736650] Lustre: 188992:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788759352/real 1788759352] req@0000000030b0a8e2 x1875649665079424/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788759368 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6778.846524] LustreError: 199876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759371 with bad export cookie 9395459191453225628 [ 6778.860358] LustreError: 199876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6778.871647] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6779.411787] Lustre: server umount lustre-MDT0001 complete [ 6808.932646] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:36:39 (1788759399) [ 6827.551818] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 6840.144369] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6880.175717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6880.559587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6880.616929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6880.748713] Lustre: lustre-MDT0000: new disk, initializing [ 6880.904960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6880.941820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6886.410901] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6897.994784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.090919] Lustre: 207982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6898.141366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6898.145098] Lustre: Skipped 1 previous similar message [ 6898.213307] Lustre: lustre-MDT0001: new disk, initializing [ 6898.367941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6898.380296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6903.824309] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6915.008711] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6922.050212] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6929.595900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6929.844681] Lustre: lustre-OST0000: new disk, initializing [ 6929.848587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6929.855692] Lustre: 209953:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6937.404748] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6939.811224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6939.829028] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6939.977601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6948.363705] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.428948] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6954.688709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6960.261585] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6960.452518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6965.730032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6965.730886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.736973] LustreError: Skipped 1 previous similar message [ 6965.738553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6965.770605] Lustre: Skipped 1 previous similar message [ 6969.195357] Lustre: server umount lustre-OST0000 complete [ 6979.730976] Lustre: server umount lustre-MDT0000 complete [ 6981.090044] LustreError: 207994:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.115064] LustreError: 207994:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 14 previous similar messages [ 6983.606914] LustreError: 208778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759576 with bad export cookie 9395459191453227420 [ 6983.609771] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6983.629328] LustreError: 208778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6983.993579] Lustre: server umount lustre-MDT0001 complete [ 6992.316159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.833665] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7005.734278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.206163] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7022.057892] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7028.717579] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7036.291930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7043.851595] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7054.027779] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7059.778889] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7060.198081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7066.324765] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7066.627123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7072.742728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7072.747752] LustreError: Skipped 1 previous similar message [ 7075.557821] Lustre: server umount lustre-OST0000 complete [ 7083.454387] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7089.655784] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7120.249260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7128.426753] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7137.876960] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7138.040682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7138.070886] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7138.077156] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7138.106656] Lustre: Skipped 1 previous similar message [ 7140.081250] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7148.513636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7148.526597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7148.527273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7148.538798] Lustre: Skipped 5 previous similar messages [ 7148.571701] Lustre: Skipped 7 previous similar messages [ 7151.184651] Lustre: server umount lustre-OST0000 complete [ 7161.449386] Lustre: server umount lustre-MDT0000 complete [ 7166.009152] LustreError: 214013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759758 with bad export cookie 9395459191453228302 [ 7166.016632] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7166.022186] LustreError: 214013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7166.742865] Lustre: server umount lustre-MDT0001 complete [ 7175.872402] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:42:46 (1788759766) [ 7183.594724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.445626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7184.448502] Lustre: Skipped 6 previous similar messages [ 7189.887237] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7199.497389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.514513] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7213.788232] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7221.708579] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7230.292903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.974753] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7245.880127] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7251.193248] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7255.848102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 7261.693801] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7262.017840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7269.919197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7270.257320] Lustre: lustre-OST0001: new disk, initializing [ 7270.261898] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7270.275636] Lustre: 221543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7277.654097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7277.671439] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7277.778610] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7277.864846] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7288.307480] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7293.914279] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7294.270039] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7299.996659] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7300.313758] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7303.654824] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7303.680931] LustreError: Skipped 1 previous similar message [ 7308.684897] Lustre: server umount lustre-OST0001 complete [ 7308.788758] LustreError: 220891:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7308.816599] LustreError: 220891:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 9 previous similar messages [ 7316.090316] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7316.414292] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7322.075755] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7322.416355] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7352.035386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7357.943739] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7357.961598] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7357.967365] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7357.988312] Lustre: Skipped 1 previous similar message [ 7361.350565] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7370.347696] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7372.512859] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7381.011329] Lustre: server umount lustre-OST0001 complete [ 7409.122638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7409.132256] Lustre: Skipped 10 previous similar messages [ 7414.240226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7414.524567] Lustre: server umount lustre-OST0000 complete [ 7419.362864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.378278] Lustre: Skipped 9 previous similar messages [ 7425.312256] Lustre: server umount lustre-MDT0000 complete [ 7429.208168] LustreError: 221541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760021 with bad export cookie 9395459191453229268 [ 7429.216216] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7429.231314] LustreError: 221541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7435.829873] Lustre: server umount lustre-MDT0001 complete [ 7462.260129] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:47:32 (1788760052) [ 7469.811383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.231682] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7470.280658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7476.359817] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7485.829572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7485.952098] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7485.999478] Lustre: 226791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7491.859851] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7500.342030] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7507.165503] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7514.920372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7515.123467] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7523.037905] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7531.765788] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7542.149772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7542.230972] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7549.707090] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7559.146872] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7561.178154] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7567.040550] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7567.397750] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7572.864379] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7573.206696] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7580.318179] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90fac3763000.ost_server_uuid 50 [ 7582.194905] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90fac3763000.ost_server_uuid in IDLE state after 0 sec [ 7583.520643] Lustre: Permanently deactivating lustre-OST0001 [ 7583.554784] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7587.324090] Lustre: Permanently reactivating lustre-OST0001 [ 7592.940378] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7592.956801] LustreError: Skipped 3 previous similar messages [ 7606.240151] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7606.358271] Lustre: server umount lustre-OST0001 complete [ 7616.757228] Lustre: server umount lustre-OST0000 complete [ 7628.106868] Lustre: server umount lustre-MDT0000 complete [ 7632.407703] LustreError: 226782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760225 with bad export cookie 9395459191453230556 [ 7632.415822] LustreError: 226782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7632.419105] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7632.667683] Lustre: server umount lustre-MDT0001 complete [ 7660.852686] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:50:50 (1788760250) [ 7670.105738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.415956] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7670.512490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7670.527960] Lustre: Skipped 5 previous similar messages [ 7674.854065] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7684.236378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.451115] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7684.492957] Lustre: 233493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7690.893653] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7699.569579] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7707.125284] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7714.382415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7714.660257] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7714.682719] Lustre: Permanently deactivating lustre-OST0000 [ 7714.694900] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7714.873671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7714.885667] Lustre: Skipped 10 previous similar messages [ 7720.067203] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7722.242071] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7729.910678] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7736.922615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7737.111704] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7745.266170] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7746.309174] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7754.212864] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7756.100770] Lustre: Permanently reactivating lustre-OST0000 [ 7756.162386] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7756.174235] Lustre: Skipped 2 previous similar messages [ 7764.690127] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7764.692766] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 000000003555f0d9, cur 1788760357 deadline 1788760342 last 1788760312 [ 7764.717443] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7764.806147] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7764.912304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7778.771509] Lustre: server umount lustre-OST0001 complete [ 7797.217843] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7797.455676] Lustre: server umount lustre-OST0000 complete [ 7808.491986] Lustre: server umount lustre-MDT0000 complete [ 7812.433686] LustreError: 233485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760405 with bad export cookie 9395459191453232222 [ 7812.453399] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.876868] Lustre: server umount lustre-MDT0001 complete [ 7826.826622] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7830.257835] Key type lgssc unregistered [ 7830.695613] LNet: 238370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7830.714098] LNetError: 238370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7830.740899] LNet: Removed LNI 192.168.202.156@tcp [ 7831.935340] Key type .llcrypt unregistered [ 7831.939938] Key type ._llcrypt unregistered [ 7857.680431] Key type ._llcrypt registered [ 7857.682359] Key type .llcrypt registered [ 7857.774945] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7858.674383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7858.881091] alg: No test for adler32 (adler32-zlib) [ 7860.089919] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 7860.392820] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7862.104221] Key type lgssc registered [ 7863.574280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7874.248602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7874.259953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.897815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7881.974428] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7885.288248] LustreError: 239836:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.300369] LustreError: 239836:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7890.412050] LustreError: 239837:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7892.761791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.057931] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7893.162446] Lustre: 239823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7893.193446] Lustre: Permanently deactivating lustre-MDT0001 [ 7893.241519] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7893.888711] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7893.900209] LustreError: 239867:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7893.929057] LustreError: 239867:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7893.949269] LustreError: 239867:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7893.982231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7899.997426] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7906.781668] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7913.205624] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7920.351282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7920.548638] LustreError: 241679:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.577316] LustreError: 241679:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7920.776515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7926.262831] LustreError: 241682:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7926.292409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7928.074405] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7931.394091] LustreError: 241682:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.379965] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7941.793508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7942.017920] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7945.092788] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7945.109825] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7948.935941] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7957.230428] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7960.224618] Lustre: Permanently reactivating lustre-MDT0001 [ 7960.240466] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7960.244401] Lustre: Skipped 2 previous similar messages [ 7964.874606] Lustre: lustre-MDT0001: Client f637516f-935d-4e1b-a94c-3407c4520d3b (at 192.168.202.56@tcp) reconnecting [ 7971.857890] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7973.555887] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7979.545602] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7979.895406] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7985.168790] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7985.515289] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7988.718311] Lustre: Permanently deactivating lustre-MDT0001 [ 7988.746027] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7988.756029] Lustre: Skipped 2 previous similar messages [ 7994.953157] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8006.626608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8006.636725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.652174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8008.677432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8008.679305] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.694025] Lustre: Skipped 1 previous similar message [ 8008.714317] Lustre: Skipped 1 previous similar message [ 8012.446232] Lustre: server umount lustre-MDT0000 complete [ 8013.811833] LustreError: 239837:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.827043] LustreError: 239837:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 8016.484128] LustreError: 239815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760609 with bad export cookie 8689007818064548830 [ 8016.505124] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8016.505313] LustreError: 239815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8016.800591] Lustre: server umount lustre-MDT0001 complete [ 8022.641790] Lustre: server umount lustre-OST0000 complete [ 8034.755779] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:57:05 (1788760625) [ 8045.965397] Lustre: server umount lustre-OST0001 complete [ 8053.549725] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 8064.339721] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8105.012369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.245907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8105.251839] Lustre: Skipped 2 previous similar messages [ 8105.297538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8105.372536] Lustre: lustre-MDT0000: new disk, initializing [ 8105.476441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8105.509499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8110.908741] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8123.057928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8123.235381] Lustre: 248125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8123.292825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8123.296912] Lustre: Skipped 1 previous similar message [ 8123.364491] Lustre: lustre-MDT0001: new disk, initializing [ 8123.450817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8123.494375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8123.528397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8130.205560] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8142.955701] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8149.102780] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8156.408380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8156.834556] Lustre: lustre-OST0000: new disk, initializing [ 8156.840061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8156.861789] Lustre: 250090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8157.020759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8163.869976] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8164.947168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8164.961871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8165.030044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8173.447213] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8176.043404] LustreError: 248129:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8185.416852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8185.583623] Lustre: lustre-OST0001: new disk, initializing [ 8185.586810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8185.595527] Lustre: 251204:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8186.050483] LustreError: 248129:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8186.080151] LustreError: 248130:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8193.960348] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8195.719843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8195.731082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8195.811025] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8196.146442] LustreError: 248130:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8203.992497] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8205.703198] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8211.425318] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8211.432357] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.438255] LustreError: Skipped 1 previous similar message [ 8211.449438] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8211.458436] Lustre: Skipped 2 previous similar messages [ 8211.477427] Lustre: Skipped 1 previous similar message [ 8216.553468] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8216.556572] Lustre: Skipped 1 previous similar message [ 8221.670969] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8221.684964] Lustre: Skipped 1 previous similar message [ 8222.179220] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8222.400137] Lustre: server umount lustre-OST0001 complete [ 8226.796793] LustreError: 251639:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.812484] LustreError: 251639:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 8242.148431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8242.167290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8242.182882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8242.194867] Lustre: Skipped 1 previous similar message [ 8245.713972] Lustre: server umount lustre-OST0000 complete [ 8251.361263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8251.369356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.392486] Lustre: Skipped 1 previous similar message [ 8255.960933] Lustre: server umount lustre-MDT0000 complete [ 8259.916455] LustreError: 248117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760852 with bad export cookie 8689007818064551427 [ 8259.930436] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8259.934490] LustreError: 248117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8274.400447] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8274.817444] Lustre: server umount lustre-MDT0001 complete [ 8286.970636] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8289.604550] Key type lgssc unregistered [ 8289.853749] LNet: 253250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8289.858446] LNetError: 253250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8289.869989] LNet: Removed LNI 192.168.202.156@tcp [ 8290.692359] Key type .llcrypt unregistered [ 8290.694237] Key type ._llcrypt unregistered [ 8334.953548] Key type ._llcrypt registered [ 8334.956866] Key type .llcrypt registered [ 8335.285691] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8337.638224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8337.697505] alg: No test for adler32 (adler32-zlib) [ 8339.084731] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 8339.545108] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8341.481247] Key type lgssc registered [ 8343.061859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8353.985572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8354.014824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8355.347161] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8355.371832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8355.613786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.562509] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8370.746768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.823739] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8370.848563] Lustre: 255457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8371.169442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8376.988448] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8386.028842] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8392.541931] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8399.930892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8400.154506] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8400.380687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.032430] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8409.347251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8414.416797] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8471.013858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8471.019022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.041528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8471.047407] Lustre: Skipped 1 previous similar message [ 8476.131552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8476.135779] Lustre: Skipped 2 previous similar messages [ 8476.940817] Lustre: server umount lustre-OST0000 complete [ 8482.555834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8495.624894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8495.818821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8501.996187] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8502.033104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8502.046247] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8502.074217] Lustre: Skipped 1 previous similar message [ 8503.617881] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8512.276960] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8553.442869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8553.453480] LustreError: Skipped 1 previous similar message [ 8553.457580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8553.468907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8553.482572] Lustre: Skipped 1 previous similar message [ 8553.501494] Lustre: Skipped 1 previous similar message [ 8558.571503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8558.583069] Lustre: Skipped 1 previous similar message [ 8558.994986] Lustre: server umount lustre-OST0000 complete [ 8565.731489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8565.735562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8565.754139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8568.816642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.314549] Lustre: server umount lustre-MDT0000 complete [ 8575.538355] LustreError: 255451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761168 with bad export cookie 5757764150534939266 [ 8575.560341] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8575.563101] LustreError: 255451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8576.131149] Lustre: server umount lustre-MDT0001 complete [ 8590.479423] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8594.007781] Key type lgssc unregistered [ 8594.338603] LNet: 260885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8594.348310] LNetError: 260885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8595.432438] LNet: Removed LNI 192.168.202.156@tcp [ 8596.500125] Key type .llcrypt unregistered [ 8596.501816] Key type ._llcrypt unregistered [ 8626.228618] Key type ._llcrypt registered [ 8626.231185] Key type .llcrypt registered [ 8626.403495] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8627.602635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8627.761950] alg: No test for adler32 (adler32-zlib) [ 8629.121247] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 8629.347703] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8631.041295] Key type lgssc registered [ 8632.439790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8640.776372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8640.812466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.285649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.212416] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8651.785381] LustreError: 262299:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8651.813548] LustreError: 262299:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8656.755756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.868142] LustreError: 262298:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.147304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8662.354487] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8670.413910] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8677.884482] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8687.253322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8687.686997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8696.277317] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8705.848405] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8712.260345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8737.768325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8737.772663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8737.779688] LustreError: Skipped 1 previous similar message [ 8737.788784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8737.809384] Lustre: Skipped 1 previous similar message [ 8742.883431] LustreError: 264125:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8743.399989] Lustre: server umount lustre-OST0000 complete [ 8749.540186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8749.564830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.591385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8749.598172] Lustre: Skipped 1 previous similar message [ 8753.129175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8753.143373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.487630] Lustre: server umount lustre-MDT0000 complete [ 8758.250311] LustreError: 262293:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8758.274778] LustreError: 262293:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8758.777541] LustreError: 265854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761351 with bad export cookie 9406381302849469333 [ 8758.783314] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8759.342111] Lustre: server umount lustre-MDT0001 complete [ 8773.064073] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8776.450829] Key type lgssc unregistered [ 8776.832854] LNet: 266605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8776.847422] LNetError: 266605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8776.884269] LNet: Removed LNI 192.168.202.156@tcp [ 8778.148466] Key type .llcrypt unregistered [ 8778.150422] Key type ._llcrypt unregistered [ 8800.442627] Key type ._llcrypt registered [ 8800.449247] Key type .llcrypt registered [ 8800.591731] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8802.457960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8802.575363] alg: No test for adler32 (adler32-zlib) [ 8803.776896] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 8804.145033] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8805.873430] Key type lgssc registered [ 8807.132188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8816.499918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8816.528331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8818.161873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8823.744234] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8827.368584] LustreError: 267825:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8827.392979] LustreError: 267825:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8832.482152] LustreError: 267826:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8834.030527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8834.440062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8840.066414] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8848.624875] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8855.299735] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8863.184460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8863.637467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8871.240514] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8881.820346] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8888.258281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8905.698167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8905.703284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8905.723565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8908.774815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8908.780973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8908.805767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8910.412544] Lustre: server umount lustre-OST0000 complete [ 8916.476023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8916.488904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.505750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8916.517420] Lustre: Skipped 1 previous similar message [ 8919.010664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8919.011913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.821498] Lustre: server umount lustre-MDT0000 complete [ 8924.133595] LustreError: 267825:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8924.149955] LustreError: 267825:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8926.358036] LustreError: 267806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761519 with bad export cookie 6912256043410085794 [ 8926.362100] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8926.367368] LustreError: 267806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8927.183794] Lustre: server umount lustre-MDT0001 complete [ 8943.416433] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8946.880633] Key type lgssc unregistered [ 8947.390575] LNet: 271528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8947.404207] LNetError: 271528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8947.418567] LNet: Removed LNI 192.168.202.156@tcp [ 8948.826226] Key type .llcrypt unregistered [ 8948.829072] Key type ._llcrypt unregistered [ 8979.817794] Key type ._llcrypt registered [ 8979.819508] Key type .llcrypt registered [ 8979.961790] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8981.262954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8981.320348] alg: No test for adler32 (adler32-zlib) [ 8982.707411] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 8983.136920] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8984.808382] Key type lgssc registered [ 8986.018888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8996.141551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8996.164154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.746292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9002.807417] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9008.099064] LustreError: 272945:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9008.130331] LustreError: 272945:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 9013.220556] LustreError: 272946:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9013.850755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9014.414095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9020.292765] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9029.629811] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9036.246545] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9044.475099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9044.852036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9052.426028] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9060.872511] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9067.379338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9093.091983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9093.098181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9093.106311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9093.127644] Lustre: Skipped 1 previous similar message [ 9098.218200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9098.232900] Lustre: Skipped 2 previous similar messages [ 9098.497339] Lustre: server umount lustre-OST0000 complete [ 9103.332741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9103.353981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9103.358576] Lustre: Skipped 1 previous similar message [ 9108.453958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9108.476577] Lustre: Skipped 1 previous similar message [ 9109.776087] Lustre: server umount lustre-MDT0000 complete [ 9113.573772] LustreError: 272945:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9113.609785] LustreError: 272945:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9114.322952] LustreError: 272927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761707 with bad export cookie 4221259160696494917 [ 9114.343613] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9114.878641] Lustre: server umount lustre-MDT0001 complete [ 9129.106703] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9132.441244] Key type lgssc unregistered [ 9132.906183] LNet: 277253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9132.920424] LNetError: 277253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9132.948150] LNet: Removed LNI 192.168.202.156@tcp [ 9134.103779] Key type .llcrypt unregistered [ 9134.110394] Key type ._llcrypt unregistered [ 9158.623364] Key type ._llcrypt registered [ 9158.632164] Key type .llcrypt registered [ 9158.780612] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9160.340146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9160.633441] alg: No test for adler32 (adler32-zlib) [ 9161.822691] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 9162.130349] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9163.921126] Key type lgssc registered [ 9165.385439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9175.512682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9175.544737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.174103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9183.500834] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9186.275786] LustreError: 278488:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9186.293327] LustreError: 278488:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 9191.394091] LustreError: 278490:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9193.404600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9193.731822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9199.486436] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9209.390452] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9217.202029] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9225.939908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9226.596742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9233.842773] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9242.411442] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9247.151386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9262.570972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9262.574679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9262.591187] Lustre: Skipped 1 previous similar message [ 9262.606977] Lustre: Skipped 1 previous similar message [ 9266.644553] Lustre: server umount lustre-OST0000 complete [ 9272.805505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9272.811448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9272.839125] Lustre: Skipped 1 previous similar message [ 9272.856159] Lustre: Skipped 1 previous similar message [ 9277.737600] Lustre: server umount lustre-MDT0000 complete [ 9277.921266] LustreError: 278488:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9277.941509] LustreError: 278488:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9282.246958] LustreError: 278455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761874 with bad export cookie 14547397043430618772 [ 9282.250256] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9282.259540] LustreError: 278455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9282.831786] Lustre: server umount lustre-MDT0001 complete [ 9297.628916] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9300.941739] Key type lgssc unregistered [ 9301.424593] LNet: 282192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9301.433690] LNetError: 282192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9301.464456] LNet: Removed LNI 192.168.202.156@tcp [ 9302.434441] Key type .llcrypt unregistered [ 9302.438574] Key type ._llcrypt unregistered [ 9316.728415] Key type ._llcrypt registered [ 9316.730691] Key type .llcrypt registered [ 9316.898628] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9339.140924] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9357.598681] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9358.898857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9358.965393] alg: No test for adler32 (adler32-zlib) [ 9360.293151] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 9360.481210] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9362.184958] Key type lgssc registered [ 9363.965772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9403.157041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9403.213594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.579250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9404.625569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9404.757333] Lustre: lustre-MDT0000: new disk, initializing [ 9404.890699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9404.946094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9409.345833] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9423.172403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9423.354658] Lustre: 285749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9423.409751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9423.416086] Lustre: Skipped 1 previous similar message [ 9423.460201] Lustre: lustre-MDT0001: new disk, initializing [ 9423.502346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9423.519867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9423.528966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9429.279252] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9440.030627] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9446.283135] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9454.328784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9454.718652] Lustre: lustre-OST0000: new disk, initializing [ 9454.726779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9454.736050] Lustre: 287718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9454.907751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9460.852027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9460.861797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9461.065501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9461.899314] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9471.924336] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9477.744817] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9478.023285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9483.326939] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9483.596250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9486.818617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9486.827878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9486.858735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9486.864955] Lustre: Skipped 1 previous similar message [ 9491.869500] Lustre: server umount lustre-OST0000 complete [ 9497.058565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9497.073817] Lustre: Skipped 1 previous similar message [ 9497.078206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9502.181887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9502.193876] Lustre: Skipped 2 previous similar messages [ 9506.277164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9506.289207] Lustre: Skipped 1 previous similar message [ 9510.368129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9510.595448] Lustre: server umount lustre-MDT0000 complete [ 9514.580619] LustreError: 287717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762107 with bad export cookie 9892183559349752779 [ 9514.591734] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9514.594235] LustreError: 287717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9514.943498] Lustre: server umount lustre-MDT0001 complete [ 9523.551978] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:21:53 (1788762113) [ 9530.433496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9531.065738] LustreError: 290130:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.089257] LustreError: 290130:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 9531.191774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9536.337907] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9536.487660] LustreError: 290130:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9540.579656] LustreError: 290131:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9544.069872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.367786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9549.040779] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9558.392943] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9564.563926] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9572.812307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9573.250509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9580.402738] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9588.649471] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9606.116623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9606.127917] LustreError: Skipped 1 previous similar message [ 9606.129474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9606.137315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9606.166892] Lustre: Skipped 2 previous similar messages [ 9606.195351] Lustre: Skipped 1 previous similar message [ 9608.398949] Lustre: server umount lustre-OST0000 complete [ 9616.354349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9616.364700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9616.389893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9616.416578] LustreError: Skipped 1 previous similar message [ 9620.879345] Lustre: server umount lustre-MDT0000 complete [ 9621.479180] LustreError: 290131:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9621.488243] LustreError: 290131:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9625.160380] LustreError: 290866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762217 with bad export cookie 9892183559349753661 [ 9625.162509] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9625.164848] LustreError: 290866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9625.767924] Lustre: server umount lustre-MDT0001 complete [ 9637.455773] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9640.336722] Key type lgssc unregistered [ 9640.699850] LNet: 293689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9640.709289] LNetError: 293689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9640.724108] LNet: Removed LNI 192.168.202.156@tcp [ 9641.640667] Key type .llcrypt unregistered [ 9641.642641] Key type ._llcrypt unregistered [ 9671.277912] Key type ._llcrypt registered [ 9671.279893] Key type .llcrypt registered [ 9671.413685] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9688.512978] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9690.036297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9690.417700] alg: No test for adler32 (adler32-zlib) [ 9692.062450] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 9692.296237] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9693.960433] Key type lgssc registered [ 9695.882099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9747.960647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9747.990081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.355178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9749.390695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9749.502513] Lustre: lustre-MDT0000: new disk, initializing [ 9749.626459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9749.661664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9755.791787] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9767.664427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9767.820803] Lustre: 297515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9767.894096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9767.898816] Lustre: Skipped 1 previous similar message [ 9767.996393] Lustre: lustre-MDT0001: new disk, initializing [ 9768.202350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9768.270524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9768.288680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9774.270535] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9784.458790] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9790.399600] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9797.168421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9797.518854] Lustre: lustre-OST2710: new disk, initializing [ 9797.523408] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9797.531649] Lustre: 299484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9797.660421] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9803.351560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9803.368539] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9803.536065] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9804.437342] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9814.369863] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9821.926787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9822.166672] Lustre: lustre-OST03e8: new disk, initializing [ 9822.178906] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9822.187431] Lustre: 300548:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9822.385092] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9827.423561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9827.429670] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9827.525232] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9828.815584] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9835.561074] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9842.124970] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9842.337661] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9847.325576] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9847.619792] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9852.218162] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9852.399645] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9857.427142] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9857.614449] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9863.657095] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9863.661344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9863.665737] Lustre: Skipped 1 previous similar message [ 9863.682031] Lustre: Skipped 3 previous similar messages [ 9864.120367] Lustre: server umount lustre-MDT0000 complete [ 9867.564358] LustreError: 297505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762460 with bad export cookie 3668034131204808398 [ 9867.584099] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9867.590751] LustreError: 297505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9867.996334] Lustre: server umount lustre-MDT0001 complete [ 9882.756396] Lustre: server umount lustre-OST2710 complete [ 9885.152436] Lustre: 295308:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788762461/real 1788762461] req@000000003c7bd5ab x1875653201823232/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788762477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9885.179291] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9885.188096] Lustre: Skipped 2 previous similar messages [ 9886.683431] Lustre: server umount lustre-OST03e8 complete [ 9906.945581] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9921.944760] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9958.644939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9958.844240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9958.871829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9958.930625] Lustre: lustre-MDT0000: new disk, initializing [ 9959.028980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9959.045331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9963.073967] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9973.651127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.813620] Lustre: 305812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9973.893313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9973.898950] Lustre: Skipped 1 previous similar message [ 9973.930445] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9973.991681] Lustre: lustre-MDT0001: new disk, initializing [ 9974.090515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9974.145349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9974.164500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9979.153292] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9988.064975] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9994.518810] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10004.398279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10004.647997] Lustre: lustre-OST0000: new disk, initializing [10004.652732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10004.659319] Lustre: 307780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10004.755434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10012.608219] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10014.836996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10014.844318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10014.968940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10023.622120] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10029.675500] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10029.927677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10034.837206] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10035.118790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10040.809562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10040.810885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10040.821543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10040.822235] Lustre: Skipped 1 previous similar message [10040.841381] Lustre: Skipped 1 previous similar message [10043.497846] Lustre: server umount lustre-OST0000 complete [10051.041789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10051.045740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10051.058295] Lustre: Skipped 1 previous similar message [10051.066500] Lustre: Skipped 1 previous similar message [10053.741088] Lustre: server umount lustre-MDT0000 complete [10057.652754] LustreError: 306606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762650 with bad export cookie 3668034131204810078 [10057.669543] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10057.680189] LustreError: 306606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10057.950921] Lustre: server umount lustre-MDT0001 complete [10068.738726] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:30:59 (1788762659) [10070.567741] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10073.137919] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:31:03 (1788762663) [10079.051417] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10112.990209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10113.319269] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10113.351177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10113.527116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10119.263750] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10129.378613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10129.484050] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10129.508160] Lustre: 311641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10129.589700] LustreError: 311653:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.611361] LustreError: 311653:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10136.356867] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10160.429051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10160.624094] Lustre: Denying initial registration attempt from nid 192.168.202.156@tcp, specified as failover [10160.629823] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10160.634549] LustreError: 313097:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [10160.639384] LustreError: 313097:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10160.641506] LustreError: 313097:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10160.704261] Lustre: server umount lustre-OST0000 complete [10160.705991] LustreError: 313097:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10170.864865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.875562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.875805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10170.881748] Lustre: Skipped 1 previous similar message [10170.917084] Lustre: Skipped 1 previous similar message [10172.865477] Lustre: server umount lustre-MDT0000 complete [10175.970361] LustreError: 311653:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10175.995671] LustreError: 311653:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [10176.627028] LustreError: 311635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762769 with bad export cookie 3668034131204810960 [10176.628365] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10176.636825] LustreError: 311635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10177.018189] Lustre: server umount lustre-MDT0001 complete [10187.506541] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10190.268964] Key type lgssc unregistered [10190.645244] LNet: 314218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10190.652502] LNetError: 314218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10191.725133] LNet: Removed LNI 192.168.202.156@tcp [10192.800278] Key type .llcrypt unregistered [10192.802603] Key type ._llcrypt unregistered [10206.185940] Key type ._llcrypt registered [10206.188185] Key type .llcrypt registered [10206.304215] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10207.793681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10207.818645] alg: No test for adler32 (adler32-zlib) [10208.887943] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [10209.115690] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10210.753129] Key type lgssc registered [10212.132864] Lustre: Echo OBD driver; http://www.lustre.org/ [10261.825864] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10270.361515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10270.375712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.656662] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10271.687143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10271.919150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10276.604463] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10285.182182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10285.285425] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10285.308726] Lustre: 316982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10285.674233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10290.222834] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10298.083662] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10303.890278] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10323.372232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.546412] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10323.709963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10329.871417] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10337.925994] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10343.905563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10343.911204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10343.927238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10344.427755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10344.442365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10344.455341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10344.460393] Lustre: Skipped 1 previous similar message [10346.497110] Lustre: server umount lustre-OST0000 complete [10352.097597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10352.105280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10352.120119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.660048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.666941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10359.780159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10359.789207] Lustre: Skipped 2 previous similar messages [10364.384583] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10364.851210] Lustre: server umount lustre-MDT0000 complete [10370.024476] LustreError: 316990:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10370.049259] LustreError: 316990:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [10370.454021] LustreError: 316977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762963 with bad export cookie 12694181594708237798 [10370.454708] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10370.466405] LustreError: 316977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10371.093766] Lustre: server umount lustre-MDT0001 complete [10383.100692] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10385.608262] Key type lgssc unregistered [10385.820580] LNet: 320558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10385.829560] LNetError: 320558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10385.845299] LNet: Removed LNI 192.168.202.156@tcp [10386.509846] Key type .llcrypt unregistered [10386.512907] Key type ._llcrypt unregistered [10411.312421] Key type ._llcrypt registered [10411.316497] Key type .llcrypt registered [10411.443594] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10413.052322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10413.142656] alg: No test for adler32 (adler32-zlib) [10414.210554] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [10414.470684] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10416.200694] Key type lgssc registered [10417.474323] Lustre: Echo OBD driver; http://www.lustre.org/ [10426.446705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10426.478318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10428.265499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10433.709267] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10437.608146] LustreError: 321972:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10437.667430] LustreError: 321972:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10442.743114] LustreError: 321973:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.265960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.819313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10450.513681] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10461.517182] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10468.784866] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10478.640415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10479.044652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10485.443218] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10495.540455] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10511.331203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10511.342943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10511.385609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10512.049370] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [10512.869294] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10512.892195] Lustre: Skipped 1 previous similar message [10515.798351] Lustre: server umount lustre-MDT0000 complete [10517.161218] LustreError: 321967:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10520.509338] LustreError: 321953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763113 with bad export cookie 4084851387664576369 [10520.516038] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10520.524217] LustreError: 321953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10522.269327] LustreError: 321968:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10522.286795] LustreError: 321968:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10522.311263] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [10522.315481] Lustre: Skipped 3 previous similar messages [10523.109799] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10525.155275] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10525.159169] Lustre: Skipped 1 previous similar message [10526.981280] Lustre: server umount lustre-MDT0001 complete [10533.268204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10544.807287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.645897] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x38b04eb1d2024e03 [10545.667495] Lustre: MGC192.168.202.156@tcp: Connection restored to 0@lo (at 0@lo) [10546.040769] LustreError: 323808:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10546.064402] LustreError: 323808:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10546.171566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10546.275171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10550.783341] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10555.374036] LustreError: 325590:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10555.394475] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10555.410314] LustreError: 325590:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10562.927923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10563.501240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10568.706203] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10570.675286] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10578.472656] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10580.007381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10586.009964] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10587.491902] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10594.276894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.280679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10594.291254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10594.294801] Lustre: Skipped 1 previous similar message [10604.522451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10604.530976] Lustre: Skipped 3 previous similar messages [10607.072375] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10607.391595] Lustre: server umount lustre-OST0000 complete [10614.753396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10614.765537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10614.803652] Lustre: Skipped 1 previous similar message [10616.873308] Lustre: server umount lustre-MDT0000 complete [10619.884520] LustreError: 325595:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10619.905638] LustreError: 325595:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [10620.671252] LustreError: 321951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763213 with bad export cookie 4084851387664584195 [10620.674149] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10620.678747] LustreError: 321951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10621.102751] Lustre: server umount lustre-MDT0001 complete [10638.448757] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10642.126837] Key type lgssc unregistered [10642.478175] LNet: 328285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10642.483601] LNetError: 328285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10642.498290] LNet: Removed LNI 192.168.202.156@tcp [10643.686173] Key type .llcrypt unregistered [10643.688514] Key type ._llcrypt unregistered [10671.203389] Key type ._llcrypt registered [10671.209308] Key type .llcrypt registered [10671.299772] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10672.550499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10672.790260] alg: No test for adler32 (adler32-zlib) [10673.779760] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [10673.987689] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10675.640826] Key type lgssc registered [10676.968553] Lustre: Echo OBD driver; http://www.lustre.org/ [10685.668416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10685.690208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10687.299798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10692.462719] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10696.722920] LustreError: 329698:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.741848] LustreError: 329698:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10701.796511] LustreError: 329699:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10702.237545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10702.882368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10708.227772] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10717.243554] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10723.425264] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10727.395200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10727.399926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10727.419528] Lustre: Skipped 1 previous similar message [10727.431552] Lustre: Skipped 1 previous similar message [10732.515570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10732.528975] Lustre: Skipped 1 previous similar message [10732.883666] Lustre: server umount lustre-MDT0000 complete [10736.766234] LustreError: 329680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763329 with bad export cookie 13682095291110489688 [10736.768828] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10736.772064] LustreError: 329680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10737.113851] Lustre: server umount lustre-MDT0001 complete [10751.630746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.927267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10751.970467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10752.164047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10757.578349] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10767.785087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10767.872241] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10767.916673] Lustre: 332285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10768.081788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10773.014371] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10782.310806] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10788.843733] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10801.076728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10801.397846] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10801.411213] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10801.418228] LustreError: 334171:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10801.426891] LustreError: 334171:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10801.434345] LustreError: 334171:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10801.561469] Lustre: server umount lustre-OST0000 complete [10801.564035] LustreError: 334171:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10815.407996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10815.688610] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10816.143223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10824.523708] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10826.069746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10833.648826] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10844.304671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10844.471281] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10844.596110] Lustre: lustre-OST0001: new disk, initializing [10844.613396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10844.628909] Lustre: 335716:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10844.774407] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10852.029748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10852.043205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10852.254955] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10853.468724] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10862.662963] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10867.681859] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10867.688268] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10867.695806] LustreError: Skipped 1 previous similar message [10867.704145] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10867.728862] Lustre: Skipped 1 previous similar message [10872.344962] Lustre: server umount lustre-OST0001 complete [10872.824427] LustreError: 334641:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.839811] LustreError: 334641:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [10877.409435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10877.412147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10877.416963] LustreError: Skipped 1 previous similar message [10877.418290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10877.428188] Lustre: Skipped 2 previous similar messages [10877.449066] Lustre: Skipped 1 previous similar message [10877.922530] LustreError: 334639:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10877.959084] LustreError: 334639:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [10883.044651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10883.054135] LustreError: 334641:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.070476] LustreError: 334641:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [10891.232123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10891.434882] Lustre: server umount lustre-OST0000 complete [10896.353493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10896.359919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.371769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.374938] Lustre: Skipped 3 previous similar messages [10902.023854] Lustre: server umount lustre-MDT0000 complete [10903.522702] LustreError: 332297:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10903.542356] LustreError: 332297:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10906.108252] LustreError: 332277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763498 with bad export cookie 13682095291110490248 [10906.115463] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10906.121396] LustreError: 332277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10906.444691] Lustre: server umount lustre-MDT0001 complete [10917.518095] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10920.778253] Key type lgssc unregistered [10921.097361] LNet: 337674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10921.109707] LNetError: 337674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10921.152861] LNet: Removed LNI 192.168.202.156@tcp [10922.356597] Key type .llcrypt unregistered [10922.358832] Key type ._llcrypt unregistered [10998.571074] Key type ._llcrypt registered [10998.573981] Key type .llcrypt registered [10998.745126] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [11017.960910] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11018.984849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11019.066978] alg: No test for adler32 (adler32-zlib) [11020.304373] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [11020.656931] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [11022.352232] Key type lgssc registered [11023.651408] Lustre: Echo OBD driver; http://www.lustre.org/ [11058.696950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11058.738343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11060.152457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11060.204156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11060.348362] Lustre: lustre-MDT0000: new disk, initializing [11060.532497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11060.572200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11066.502273] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11077.286313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.419486] Lustre: 342604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11077.475357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11077.478635] Lustre: Skipped 1 previous similar message [11077.560940] Lustre: lustre-MDT0001: new disk, initializing [11077.725260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11077.787254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11077.802983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11082.708665] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11093.435910] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11100.035137] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11108.271959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11108.559809] Lustre: lustre-OST0000: new disk, initializing [11108.563320] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11108.572898] Lustre: 344575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11108.655254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11116.783650] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11117.091764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11117.100306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11117.208660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11126.908468] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11132.009894] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11132.345872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11136.352645] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11136.574049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11142.627461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11142.659680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11142.671692] Lustre: Skipped 1 previous similar message [11147.747753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11147.764684] Lustre: Skipped 1 previous similar message [11152.871610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11152.875096] Lustre: Skipped 1 previous similar message [11153.376678] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11153.831142] Lustre: server umount lustre-OST0000 complete [11159.521618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11159.535879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11159.551951] Lustre: Skipped 1 previous similar message [11159.569675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11163.105477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11165.883478] Lustre: server umount lustre-MDT0000 complete [11169.310725] LustreError: 342598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763762 with bad export cookie 3572313730548052329 [11169.326326] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11169.969329] Lustre: server umount lustre-MDT0001 complete [11179.495771] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:49:29 (1788763769) [11188.417542] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:49:39 (1788763779) [11197.668241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11198.351236] LustreError: 347331:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.375384] LustreError: 347331:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [11198.426391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11203.553098] LustreError: 347332:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11204.477271] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11208.675969] LustreError: 347331:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11213.800247] LustreError: 347332:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11213.973083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11214.350303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11220.114313] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11229.609472] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11237.416688] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11244.919659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.454878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11253.485754] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11261.983403] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11280.966341] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11283.163224] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11285.143968] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11286.802713] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11289.032409] Lustre: Failing over lustre-MDT0000 [11289.290922] LustreError: 347326:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11289.554117] Lustre: server umount lustre-MDT0000 complete [11289.580515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11289.593083] Lustre: Skipped 1 previous similar message [11299.489178] LustreError: 347328:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11299.530350] LustreError: 347328:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [11308.149848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.343729] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11308.576813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11308.630245] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11309.734374] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11313.035187] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11313.656830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11313.704241] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11313.737489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11324.964870] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11327.906093] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11332.776504] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11335.463647] Lustre: Failing over lustre-MDT0000 [11336.031776] Lustre: server umount lustre-MDT0000 complete [11339.237588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11339.239061] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11339.244304] LustreError: 348743:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11339.244316] LustreError: 348743:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [11339.300681] Lustre: Skipped 3 previous similar messages [11354.595295] Lustre: 340827:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788763931/real 1788763931] req@000000000f45346a x1875654594363904/t0(0) o400->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788763947 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11354.609802] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11356.437842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11365.208898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11365.292227] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11366.078426] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11370.489345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11370.498494] Lustre: Skipped 2 previous similar messages [11370.500566] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11370.537182] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11370.619969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11380.732398] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11382.557300] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11390.953666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11390.954113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11390.962345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11390.962355] Lustre: Skipped 2 previous similar messages [11390.974550] Lustre: Skipped 1 previous similar message [11395.138602] Lustre: server umount lustre-MDT0000 complete [11396.069497] LustreError: 347326:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11396.100434] LustreError: 347326:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 25 previous similar messages [11398.778321] LustreError: 347312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763991 with bad export cookie 3572313730548055045 [11398.782945] LustreError: 347312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11398.789763] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11399.107967] Lustre: server umount lustre-MDT0001 complete [11413.425756] Lustre: server umount lustre-OST0000 complete [11429.822251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.458344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11435.334591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11444.657657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11450.230151] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11458.572876] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11465.572876] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11472.775187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11476.217041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11480.169163] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11488.437127] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11500.017312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11500.018306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11500.034282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11500.034293] Lustre: Skipped 2 previous similar messages [11500.069196] Lustre: Skipped 2 previous similar messages [11505.106355] Lustre: server umount lustre-OST0000 complete [11515.373684] LustreError: 353690:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11515.415489] LustreError: 353690:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [11515.798992] Lustre: server umount lustre-MDT0000 complete [11520.105043] LustreError: 356474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764112 with bad export cookie 3572313730548055787 [11520.107470] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11520.124963] LustreError: 356474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11520.587760] Lustre: server umount lustre-MDT0001 complete [11533.603418] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11536.960408] Key type lgssc unregistered [11537.428873] LNet: 357228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11537.438530] LNetError: 357228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11537.455269] LNet: Removed LNI 192.168.202.156@tcp [11538.299496] Key type .llcrypt unregistered [11538.301522] Key type ._llcrypt unregistered [11565.201778] Key type ._llcrypt registered [11565.204265] Key type .llcrypt registered [11565.299533] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11566.252894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11566.338981] alg: No test for adler32 (adler32-zlib) [11567.497779] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [11567.828805] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [11569.576336] Key type lgssc registered [11571.179719] Lustre: Echo OBD driver; http://www.lustre.org/ [11580.072687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11580.090159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11581.495505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11586.352517] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11591.651844] LustreError: 358742:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11591.673751] LustreError: 358742:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [11594.184796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11594.729265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11600.915692] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11611.117251] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11617.556057] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11624.048820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11624.533127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11626.626805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11631.301363] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11638.856119] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11650.337968] Lustre: server umount lustre-MDT0000 complete [11650.537527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11650.544912] LustreError: 361237:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11650.551475] Lustre: Skipped 1 previous similar message [11650.578320] LustreError: 361237:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [11653.793800] LustreError: 359476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764246 with bad export cookie 9806416270321257310 [11653.798065] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11653.808283] LustreError: 359476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11654.112213] Lustre: server umount lustre-MDT0001 complete [11667.619352] Lustre: server umount lustre-OST0000 complete [11683.227139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.562840] LustreError: 362454:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11683.652907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11688.877739] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11688.931347] LustreError: 362455:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.055750] LustreError: 362454:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.649255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11699.094666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11704.222375] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11712.491876] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11718.839543] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11725.561441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11725.850365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11728.958864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11732.250296] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11739.477078] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11753.446335] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11753.467419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11753.476937] Lustre: Skipped 2 previous similar messages [11753.495118] Lustre: Skipped 2 previous similar messages [11758.017052] Lustre: server umount lustre-MDT0000 complete [11758.569139] LustreError: 364296:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11758.588135] LustreError: 364296:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [11762.589549] LustreError: 362435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764355 with bad export cookie 9806416270321258430 [11762.593488] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11762.605679] LustreError: 362435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11763.053106] Lustre: server umount lustre-MDT0001 complete [11778.054099] Lustre: server umount lustre-OST0000 complete [11797.127160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.096764] LustreError: 366370:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11798.292963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11804.420238] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11813.946340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11818.497534] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11827.020037] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11833.814984] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11841.775829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11842.368145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11842.377381] Lustre: Skipped 1 previous similar message [11843.402506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11850.134151] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11860.511127] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11869.163072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11869.164260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11869.166154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11869.180029] LustreError: Skipped 1 previous similar message [11869.225614] Lustre: Skipped 2 previous similar messages [11873.380518] Lustre: server umount lustre-OST0000 complete [11879.395556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11879.417333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11879.423530] Lustre: Skipped 1 previous similar message [11883.891655] Lustre: server umount lustre-MDT0000 complete [11884.519581] LustreError: 367098:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.549641] LustreError: 367098:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [11888.312220] LustreError: 366352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764481 with bad export cookie 9806416270321259452 [11888.321684] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11888.322975] LustreError: 366352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11888.720243] Lustre: server umount lustre-MDT0001 complete [11901.578882] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11904.301946] Key type lgssc unregistered [11904.532454] LNet: 369915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11904.537819] LNetError: 369915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11904.553406] LNet: Removed LNI 192.168.202.156@tcp [11905.671895] Key type .llcrypt unregistered [11905.673665] Key type ._llcrypt unregistered [11934.449100] Key type ._llcrypt registered [11934.452726] Key type .llcrypt registered [11934.666821] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11936.156986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11936.190458] alg: No test for adler32 (adler32-zlib) [11937.328164] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [11937.736769] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [11939.448352] Key type lgssc registered [11940.622366] Lustre: Echo OBD driver; http://www.lustre.org/ [11948.145143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11948.158097] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.163699] LustreError: 371357:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11948.185650] LustreError: 371357:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11948.191420] LustreError: 371357:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11948.198808] LustreError: 371357:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11948.205108] LustreError: 371357:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11957.321952] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11957.358186] LustreError: 371754:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11957.388950] LustreError: 371754:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11957.403523] LustreError: 371754:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11957.411060] LustreError: 371754:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11957.426146] LustreError: 371754:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11981.932547] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11985.190824] Key type lgssc unregistered [11985.487766] LNet: 372757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11985.494544] LNetError: 372757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11985.525593] LNet: Removed LNI 192.168.202.156@tcp [11986.573485] Key type .llcrypt unregistered [11986.578618] Key type ._llcrypt unregistered [12007.306896] Key type ._llcrypt registered [12007.308878] Key type .llcrypt registered [12007.515209] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [12024.151121] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [12025.248370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12025.340653] alg: No test for adler32 (adler32-zlib) [12026.526515] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [12026.884869] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [12028.600255] Key type lgssc registered [12029.901549] Lustre: Echo OBD driver; http://www.lustre.org/ [12068.395157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12068.436290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12069.733574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12069.770379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12069.867163] Lustre: lustre-MDT0000: new disk, initializing [12069.959884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12070.000544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12075.852480] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12086.528144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12086.675575] Lustre: 375912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12086.761323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12086.768222] Lustre: Skipped 1 previous similar message [12086.851230] Lustre: lustre-MDT0001: new disk, initializing [12086.965212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12087.021184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12087.033509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12092.421572] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12102.788487] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12110.132903] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12117.553947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.016614] Lustre: lustre-OST0000: new disk, initializing [12118.021718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12118.037278] Lustre: 377884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12118.196588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12125.265091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12125.280669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12125.334065] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12125.419900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12134.296407] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12139.687681] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12140.100356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12146.258592] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12146.730959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12151.266420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12151.281923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.288227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12151.304197] Lustre: Skipped 1 previous similar message [12155.350310] Lustre: server umount lustre-OST0000 complete [12161.527627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12161.536426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12161.549132] Lustre: Skipped 1 previous similar message [12161.562736] Lustre: Skipped 2 previous similar messages [12166.633725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12171.746235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12171.765524] Lustre: Skipped 2 previous similar messages [12174.304135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12174.583522] Lustre: server umount lustre-MDT0000 complete [12178.613480] LustreError: 375906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764771 with bad export cookie 1071601618608694125 [12178.616754] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12178.627110] LustreError: 375906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12179.080633] Lustre: server umount lustre-MDT0001 complete [12186.747797] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12188.286688] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:06:19 (1788764779) [12194.573244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12195.208353] LustreError: 380345:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.231206] LustreError: 380345:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [12195.298450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12199.968331] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12200.429899] LustreError: 380346:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12204.580343] LustreError: 380345:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12209.646200] LustreError: 380346:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12211.334721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12212.085244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12217.853708] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12225.869664] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12232.890953] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12241.195189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12241.600256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12249.011224] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12258.693331] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12265.997971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12266.233486] Lustre: lustre-OST0001: new disk, initializing [12266.272969] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12266.282743] Lustre: 383188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12266.351431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12272.035920] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12275.886896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12275.895882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12276.046907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12282.646410] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12286.433399] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12286.444087] LustreError: Skipped 1 previous similar message [12286.460100] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12286.469187] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12286.479186] Lustre: Skipped 1 previous similar message [12286.496347] Lustre: Skipped 1 previous similar message [12292.596031] Lustre: server umount lustre-OST0001 complete [12294.839288] LustreError: 382169:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12306.922584] LustreError: 382170:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12306.948925] LustreError: 382170:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [12311.521935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12311.533090] LustreError: Skipped 1 previous similar message [12311.541918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12311.558474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12311.561070] Lustre: Skipped 3 previous similar messages [12324.321144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12324.752222] Lustre: server umount lustre-OST0000 complete [12329.953508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12329.963234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12329.976136] Lustre: Skipped 1 previous similar message [12329.981197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12329.990058] Lustre: Skipped 6 previous similar messages [12334.607928] Lustre: server umount lustre-MDT0000 complete [12337.633261] LustreError: 380345:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12337.653743] LustreError: 380345:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [12338.360756] LustreError: 380327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764931 with bad export cookie 1071601618608695007 [12338.376177] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12338.380519] LustreError: 380327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12344.929922] Lustre: server umount lustre-MDT0001 complete [12357.104698] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [12360.151175] Key type lgssc unregistered [12360.528438] LNet: 385177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12360.535322] LNetError: 385177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12360.549731] LNet: Removed LNI 192.168.202.156@tcp [12361.534371] Key type .llcrypt unregistered [12361.536814] Key type ._llcrypt unregistered [12416.391569] Key type ._llcrypt registered [12416.394456] Key type .llcrypt registered [12416.558058] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [12417.996614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12418.205158] alg: No test for adler32 (adler32-zlib) [12419.431673] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [12419.831863] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [12421.640427] Key type lgssc registered [12423.211893] Lustre: Echo OBD driver; http://www.lustre.org/ [12432.545147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12432.575245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12433.959890] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12434.002475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12434.262765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12439.538260] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12449.411280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12449.625180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12449.692224] Lustre: 387974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12450.094863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12455.799816] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12466.091879] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12473.189206] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12475.884348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12475.905498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12475.944201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12481.505805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12481.522331] Lustre: Skipped 3 previous similar messages [12486.627153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12486.639694] Lustre: Skipped 1 previous similar message [12490.210074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12490.798562] Lustre: server umount lustre-MDT0000 complete [12495.425709] LustreError: 387966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788765088 with bad export cookie 18012106245215784433 [12495.439459] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12495.443985] LustreError: 387966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12495.875613] Lustre: server umount lustre-MDT0001 complete [12507.395760] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:11:37 (1788765097) [12514.762431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12515.297895] LustreError: 390488:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.316452] LustreError: 390488:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [12515.366278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12520.418806] LustreError: 390489:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12520.589347] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12525.540784] LustreError: 390488:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12529.564274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12530.051977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12534.894347] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12543.357923] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12549.121284] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12555.974883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12556.139471] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12556.352551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12563.292989] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12571.220519] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12579.021672] Lustre: Permanently deactivating lustre-OST0000 [12579.043341] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12589.126558] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12590.562294] LustreError: 393525:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12590.571610] LustreError: 393525:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12590.577318] LustreError: 393525:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12594.035596] LustreError: 393574:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12594.041696] LustreError: 393574:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12602.401802] Lustre: server umount lustre-OST0000 complete [12605.123184] LustreError: 393823:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12605.128353] LustreError: 393823:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12612.068942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12612.086933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12612.108093] Lustre: Skipped 1 previous similar message [12612.121285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12613.096691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12618.213058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12618.223167] Lustre: Skipped 3 previous similar messages [12618.353585] Lustre: server umount lustre-MDT0000 complete [12623.334127] LustreError: 391486:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12623.356361] LustreError: 391486:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12624.454338] LustreError: 392323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788765217 with bad export cookie 18012106245215785014 [12624.458731] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12624.475990] LustreError: 392323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12635.243776] Lustre: server umount lustre-MDT0001 complete [12645.361920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12651.022433] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12653.491396] LustreError: 394984:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12659.331491] Lustre: 395323:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.156@tcp [12661.422357] Lustre: server umount MGS complete [12673.014060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12673.850498] LustreError: 396046:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12673.892566] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12673.903902] Lustre: Skipped 1 previous similar message [12674.001345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12678.934167] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12684.265933] LustreError: 396047:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12684.282741] LustreError: 396047:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12687.752908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12688.068875] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12694.247733] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12702.011678] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12707.486505] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12709.188917] Lustre: Permanently reactivating lustre-OST0000 [12709.201951] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12709.208482] Lustre: Skipped 2 previous similar messages [12737.001376] LustreError: 397828:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12745.572480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12745.888464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12745.891799] Lustre: Skipped 1 previous similar message [12751.594325] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12759.706307] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12766.480133] Lustre: server umount lustre-OST0000 complete [12770.273382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12770.277100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12770.283791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12780.156171] Lustre: server umount lustre-MDT0000 complete [12783.076730] LustreError: 396046:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12783.098917] LustreError: 396046:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [12784.111093] Lustre: 387297:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788765360/real 1788765360] req@000000009b93da93 x1875656061324416/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788765376 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12784.149277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12784.162539] Lustre: Skipped 1 previous similar message [12785.136822] LustreError: 396071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788765377 with bad export cookie 18012106245215786064 [12785.153475] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12785.579749] Lustre: server umount lustre-MDT0001 complete [12800.750402] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [12804.800525] Key type lgssc unregistered [12805.487081] LNet: 400412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12805.523546] LNetError: 400412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12805.577461] LNet: Removed LNI 192.168.202.156@tcp [12807.512469] Key type .llcrypt unregistered [12807.514170] Key type ._llcrypt unregistered [12830.388725] Key type ._llcrypt registered [12830.391589] Key type .llcrypt registered [12830.479648] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [12847.654019] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [12849.157546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12849.226859] alg: No test for adler32 (adler32-zlib) [12850.443904] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [12850.805516] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [12852.488237] Key type lgssc registered [12853.606972] Lustre: Echo OBD driver; http://www.lustre.org/ [12891.505875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12891.543749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12892.869105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12892.910430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12893.027626] Lustre: lustre-MDT0000: new disk, initializing [12893.167601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12893.189061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12898.241750] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12908.530683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12908.676955] Lustre: 403568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12908.783273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12908.792871] Lustre: Skipped 1 previous similar message [12908.865376] Lustre: lustre-MDT0001: new disk, initializing [12908.950487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12908.991597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12909.007957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12914.265968] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12925.899644] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12932.441426] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12940.126802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12940.491796] Lustre: lustre-OST0000: new disk, initializing [12940.496213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12940.505924] Lustre: 405535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12940.608676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12947.748934] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12950.643184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12950.663807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12950.785247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12957.617328] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12963.099412] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12963.324220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12968.018872] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12968.218695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12971.491942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12971.492024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12971.503506] LustreError: Skipped 1 previous similar message [12971.504502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12971.533891] Lustre: Skipped 1 previous similar message [12976.612315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12976.624421] Lustre: Skipped 2 previous similar messages [12981.738223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12981.750767] Lustre: Skipped 1 previous similar message [12984.288184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12984.619885] Lustre: server umount lustre-OST0000 complete [12990.948879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12990.953270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12990.968500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12991.972416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12995.590387] Lustre: server umount lustre-MDT0000 complete [12997.091070] LustreError: 403579:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12997.105135] LustreError: 403579:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [12999.923706] LustreError: 403559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788765592 with bad export cookie 16993614207461303325 [12999.931987] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12999.934541] LustreError: 403559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13000.173494] Lustre: server umount lustre-MDT0001 complete [13009.485115] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13011.124233] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:20:01 (1788765601) [13019.374702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13019.987576] LustreError: 407999:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13020.137566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13025.249845] LustreError: 408000:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13025.279677] LustreError: 408000:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13026.906384] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13029.349965] LustreError: 407999:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13034.480260] LustreError: 408000:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13036.762478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13037.169159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13042.864883] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13050.936368] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13057.575038] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13065.756673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13066.201930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13073.704843] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13081.613490] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13083.565554] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13090.198476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13097.954239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13097.961079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13097.976353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13097.984382] Lustre: Skipped 2 previous similar messages [13100.005245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13101.722479] Lustre: server umount lustre-OST0000 complete [13105.125207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13105.139502] Lustre: Skipped 2 previous similar messages [13110.240977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13110.247284] Lustre: Skipped 4 previous similar messages [13111.493754] Lustre: server umount lustre-MDT0000 complete [13115.365317] LustreError: 407994:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13115.387377] LustreError: 407994:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13115.409448] LustreError: 407981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788765708 with bad export cookie 16993614207461304207 [13115.422736] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13115.428378] LustreError: 407981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13115.780839] Lustre: server umount lustre-MDT0001 complete [13127.174356] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [13130.260172] Key type lgssc unregistered [13130.531754] LNet: 411726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13130.544126] LNetError: 411726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13130.566326] LNet: Removed LNI 192.168.202.156@tcp [13131.573101] Key type .llcrypt unregistered [13131.574800] Key type ._llcrypt unregistered [13154.858373] Key type ._llcrypt registered [13154.861587] Key type .llcrypt registered [13154.987349] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [13155.476338] Key type .llcrypt unregistered [13155.477996] Key type ._llcrypt unregistered [13174.099204] Key type ._llcrypt registered [13174.114069] Key type .llcrypt registered [13174.261515] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13175.779790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13175.843479] alg: No test for adler32 (adler32-zlib) [13176.928236] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [13177.220520] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [13178.888217] Key type lgssc registered [13179.978249] Lustre: Echo OBD driver; http://www.lustre.org/ [13188.224852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13188.255337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13189.737918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13194.679757] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13199.845480] LustreError: 413933:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13199.889962] LustreError: 413933:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [13204.188584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13204.542090] LustreError: 413932:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13204.589455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13210.091832] LustreError: 413933:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.111784] LustreError: 413933:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [13212.560422] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13215.214897] LustreError: 413932:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13215.238519] LustreError: 413932:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13215.246287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13220.335771] LustreError: 413933:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.358775] LustreError: 413933:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13220.516427] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13228.769101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13229.492375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13236.569162] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13245.923559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13245.934367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13245.954386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13250.019248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13250.028677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13250.030062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13250.076303] Lustre: Skipped 1 previous similar message [13251.340914] Lustre: server umount lustre-OST0000 complete [13255.139220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13255.141075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13255.160439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13255.187332] Lustre: Skipped 1 previous similar message [13260.260421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13261.459353] Lustre: server umount lustre-MDT0000 complete [13265.292341] LustreError: 413915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788765858 with bad export cookie 4253512240488203334 [13265.313844] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13265.710667] Lustre: server umount lustre-MDT0001 complete [13276.457685] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [13279.196658] Key type lgssc unregistered [13279.512228] LNet: 417076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13279.518792] LNetError: 417076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13279.539824] LNet: Removed LNI 192.168.202.156@tcp [13280.491659] Key type .llcrypt unregistered [13280.494962] Key type ._llcrypt unregistered [13305.102485] Key type ._llcrypt registered [13305.114100] Key type .llcrypt registered [13305.322551] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13306.492938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13306.510984] alg: No test for adler32 (adler32-zlib) [13307.708736] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [13307.941959] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [13309.568534] Key type lgssc registered [13310.708184] Lustre: Echo OBD driver; http://www.lustre.org/ [13319.270613] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13319.301356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13334.945219] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13340.064302] LustreError: 418465:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 4: rc = -110 [13366.816253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13374.669460] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13383.966750] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13391.917729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13392.394555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13392.526966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13397.475461] LustreError: 418491:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13397.505802] LustreError: 418491:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [13398.279525] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13401.597648] LustreError: 419503:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13401.627404] LustreError: 419503:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13406.713186] LustreError: 418490:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13406.733456] LustreError: 418490:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13407.869689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13408.511420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13413.932631] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13421.189944] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13426.998144] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13438.434405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13438.439762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13438.451028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13439.461090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13439.466398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13439.476603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13439.511164] Lustre: Skipped 1 previous similar message [13441.392958] Lustre: server umount lustre-OST0000 complete [13449.699077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13449.706108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13449.706965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13449.754775] Lustre: Skipped 1 previous similar message [13451.904229] Lustre: server umount lustre-MDT0000 complete [13454.832677] LustreError: 419508:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13454.869489] LustreError: 419508:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13455.832288] LustreError: 418472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788766048 with bad export cookie 5963365987483413346 [13455.849278] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13456.298395] Lustre: server umount lustre-MDT0001 complete [13468.190407] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [13471.978920] Key type lgssc unregistered [13472.306846] LNet: 422034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13472.318459] LNetError: 422034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13472.336750] LNet: Removed LNI 192.168.202.156@tcp [13473.686513] Key type .llcrypt unregistered [13473.688658] Key type ._llcrypt unregistered [13502.942787] Key type ._llcrypt registered [13502.946275] Key type .llcrypt registered [13503.137409] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13504.266817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13504.431473] alg: No test for adler32 (adler32-zlib) [13505.684747] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [13505.953924] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [13507.614172] Key type lgssc registered [13508.806533] Lustre: Echo OBD driver; http://www.lustre.org/ [13516.333744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13516.351740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13517.785598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13522.969763] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13528.045083] LustreError: 423446:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13528.058622] LustreError: 423446:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [13531.930258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13532.428907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13537.053994] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13544.832816] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13550.295945] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13557.526874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13557.937076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13563.081293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13564.484862] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13572.409691] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13578.721527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13578.729689] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13578.740348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13578.769882] Lustre: Skipped 1 previous similar message [13580.477706] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [13580.489510] Lustre: Skipped 2 previous similar messages [13582.846148] Lustre: server umount lustre-MDT0000 complete [13583.879988] LustreError: 423446:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13583.908120] LustreError: 423446:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [13585.450260] LustreError: 423428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.56@tcp arrived at 1788766178 with bad export cookie 17374083768029331431 [13585.485490] LustreError: 423428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13588.481911] LustreError: 426024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788766181 with bad export cookie 17374083768029330535 [13588.501832] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13588.978982] LustreError: 423442:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13588.982867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13589.001062] LustreError: 423442:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13589.036687] Lustre: Skipped 1 previous similar message [13589.050571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13594.081731] LustreError: 423441:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13594.089463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13594.099399] LustreError: 423441:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13595.150218] Lustre: server umount lustre-OST0000 complete [13599.202971] LustreError: 423442:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13599.218143] LustreError: 423442:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13602.247992] Lustre: server umount lustre-MDT0001 complete [13616.109357] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [13619.396754] Key type lgssc unregistered [13619.650797] LNet: 427130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13619.656202] LNetError: 427130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13619.680740] LNet: Removed LNI 192.168.202.156@tcp [13620.602586] Key type .llcrypt unregistered [13620.604518] Key type ._llcrypt unregistered [13650.045739] Key type ._llcrypt registered [13650.048264] Key type .llcrypt registered [13650.231503] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13651.356491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13651.435837] alg: No test for adler32 (adler32-zlib) [13652.979512] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [13653.362436] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [13655.104228] Key type lgssc registered [13656.910969] Lustre: Echo OBD driver; http://www.lustre.org/ [13665.196122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13665.226124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13666.643916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13672.162400] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13677.032928] LustreError: 428547:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13677.050673] LustreError: 428547:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [13681.710223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13682.334689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13687.287495] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13695.089951] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13701.309935] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13708.164940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13708.577690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13711.673431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13715.154949] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13722.475664] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13728.738820] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13728.751099] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13728.776815] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13730.467755] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [13732.326756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13732.333025] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13733.620595] Lustre: server umount lustre-MDT0001 complete [13735.588484] LustreError: 428544:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13737.443893] LustreError: 428544:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13737.486222] LustreError: 428544:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13742.562539] LustreError: 428548:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13742.566985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13742.579712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13742.601214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13742.606296] Lustre: Skipped 1 previous similar message [13746.505250] Lustre: server umount lustre-OST0000 complete [13747.685972] LustreError: 428547:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13750.459987] Lustre: server umount lustre-MDT0000 complete [13765.855542] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [13768.796966] Key type lgssc unregistered [13769.160471] LNet: 432231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13769.171023] LNetError: 432231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13769.185228] LNet: Removed LNI 192.168.202.156@tcp [13770.088646] Key type .llcrypt unregistered [13770.091278] Key type ._llcrypt unregistered [13795.859397] Key type ._llcrypt registered [13795.861200] Key type .llcrypt registered [13795.997227] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [13810.877835] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13812.187659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13812.211909] alg: No test for adler32 (adler32-zlib) [13813.229457] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [13813.487966] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [13815.208849] Key type lgssc registered [13816.250727] Lustre: Echo OBD driver; http://www.lustre.org/ [13857.208542] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13874.579574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13874.616110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13876.014606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13876.075613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13876.206184] Lustre: lustre-MDT0000: new disk, initializing [13876.301928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13876.344481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13882.018131] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13897.095547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13897.279173] Lustre: lustre-OST0000: new disk, initializing [13897.290282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13897.296082] Lustre: Skipped 1 previous similar message [13897.303559] Lustre: 437260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13897.415536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13904.000820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13904.012466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13904.053536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13904.244260] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13913.343587] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13928.727148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13928.832724] Lustre: 436244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13928.906487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13928.969361] Lustre: lustre-MDT0001: new disk, initializing [13929.091836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13929.135869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13929.151816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13933.633959] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13944.415789] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13944.604560] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13950.092638] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13950.358390] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13957.308546] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [13959.141129] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13959.181442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13959.189325] Lustre: Skipped 1 previous similar message [13962.546509] Lustre: server umount lustre-MDT0001 complete [13967.519704] LustreError: 436249:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13967.538430] LustreError: 436249:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [13969.380071] LustreError: 436254:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13969.410265] LustreError: 436254:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13974.510381] LustreError: 436255:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13974.528652] LustreError: 436255:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13979.639662] LustreError: 437971:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13984.740189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13984.742456] LustreError: 436255:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13984.760291] Lustre: Skipped 1 previous similar message [13984.762937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13984.788678] LustreError: 436255:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [13987.843975] Lustre: server umount lustre-OST0000 complete [13998.446695] Lustre: server umount lustre-MDT0000 complete [14014.441028] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [14017.427226] Key type lgssc unregistered [14017.790905] LNet: 441010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14017.796965] LNetError: 441010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14017.815536] LNet: Removed LNI 192.168.202.156@tcp [14018.786203] Key type .llcrypt unregistered [14018.788148] Key type ._llcrypt unregistered [14061.141486] Key type ._llcrypt registered [14061.143265] Key type .llcrypt registered [14061.228402] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [14061.517482] Key type .llcrypt unregistered [14061.519943] Key type ._llcrypt unregistered [14079.950568] Key type ._llcrypt registered [14079.958780] Key type .llcrypt registered [14080.113567] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [14081.917661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14081.986094] alg: No test for adler32 (adler32-zlib) [14083.199396] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [14083.418238] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [14085.104228] Key type lgssc registered [14085.881326] Lustre: Echo OBD driver; http://www.lustre.org/ [14115.223966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14115.324785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14116.687522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14116.774834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14117.017458] Lustre: lustre-MDT0000: new disk, initializing [14117.212738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14117.253245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14123.237023] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14136.609229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14136.839468] Lustre: 444394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14136.940230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14136.952249] Lustre: Skipped 1 previous similar message [14137.113798] Lustre: lustre-MDT0001: new disk, initializing [14137.279849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14137.357215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14137.375895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14142.758824] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14154.154691] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14160.101329] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14166.624145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14167.027053] Lustre: lustre-OST0000: new disk, initializing [14167.037854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14167.053739] Lustre: 446362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14167.220656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14174.434146] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14175.849966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14175.876981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14176.017126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14183.315442] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14193.638559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14193.659103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14193.685466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14196.713210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14196.726554] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14199.144579] Lustre: server umount lustre-MDT0000 complete [14204.919256] LustreError: 444385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788766797 with bad export cookie 9026131228377739886 [14204.939911] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14204.943399] LustreError: 444385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14205.769074] Lustre: server umount lustre-MDT0001 complete [14220.846167] Lustre: server umount lustre-OST0000 complete [14234.553878] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:40:24 (1788766824) [14243.844981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14244.472923] LustreError: 448392:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14244.491628] LustreError: 448392:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [14244.561675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14249.761697] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14249.953944] LustreError: 448393:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14255.075206] LustreError: 448392:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14259.490424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14259.926827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14265.053996] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14273.865705] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14281.410274] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14302.507251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14302.758388] Lustre: Found index 0 for lustre-OST0000, updating log [14303.019079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14311.667111] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14315.386324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14322.250712] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14328.858276] Lustre: server umount lustre-OST0000 complete [14346.210382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14346.211897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14346.216955] Lustre: Skipped 2 previous similar messages [14346.234533] Lustre: Skipped 3 previous similar messages [14347.232409] Lustre: 442926:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788766923/real 1788766923] req@0000000042acfaed x1875657806129280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788766939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14347.267320] Lustre: 442926:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14351.328913] Lustre: 442928:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788766928/real 1788766928] req@00000000373c2fbe x1875657806130560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788766944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14352.354072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14352.362391] Lustre: Skipped 3 previous similar messages [14355.568735] Lustre: server umount lustre-MDT0000 complete [14356.458200] LustreError: 448389:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14356.499498] LustreError: 448389:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [14357.473023] Lustre: 442928:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788766933/real 1788766933] req@00000000733d8616 x1875657806131712/t0(0) o400->lustre-OST0000-osc-MDT0001@1.2.3.4@tcp:28/4 lens 224/224 e 0 to 1 dl 1788766949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14360.008609] LustreError: 448372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788766952 with bad export cookie 9026131228377741279 [14360.011675] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14360.032822] LustreError: 448372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14360.633857] Lustre: server umount lustre-MDT0001 complete [14370.484794] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:42:40 (1788766960) [14371.232521] LNet: 1 local NIs in recovery (showing 1): 192.168.202.156@tcp [15001.756147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15002.468285] LustreError: 454808:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15002.475158] LustreError: 454808:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [15002.542319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15007.741716] LustreError: 454809:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15008.522256] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [15012.849992] LustreError: 454808:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15014.883367] LustreError: 454809:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15014.912372] LustreError: 454809:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [15020.008188] LustreError: 454808:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15021.019604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15021.659940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15027.784702] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [15036.974717] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15046.332510] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15063.584964] LNet: 1 local NIs in recovery (showing 1): 192.168.202.156@tcp [15068.612382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15068.909261] Lustre: Found index 0 for lustre-OST0000, updating log [15069.343281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15078.105723] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [15087.286761] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15091.024222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15091.045371] Lustre: Skipped 1 previous similar message [15091.053356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15096.404788] Lustre: server umount lustre-OST0000 complete [15102.050179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15102.057789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15102.064824] Lustre: Skipped 1 previous similar message [15102.092589] Lustre: Skipped 1 previous similar message [15103.138566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15103.153774] Lustre: Skipped 1 previous similar message [15105.192259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15105.194931] Lustre: Skipped 2 previous similar messages [15107.497584] Lustre: server umount lustre-MDT0000 complete [15108.269807] LustreError: 454809:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15108.291274] LustreError: 454809:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [15112.328525] LustreError: 454789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788767705 with bad export cookie 9026131228377742364 [15112.355450] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15112.817424] Lustre: server umount lustre-MDT0001 complete [15124.001975] LNet: 1 local NIs in recovery (showing 1): 192.168.202.156@tcp [15146.998776] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [15160.216910] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [15201.454377] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:56:32 (1788767792) [15225.631670] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [15228.704414] Key type lgssc unregistered [15229.007657] LNet: 461416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15229.011957] LNetError: 461416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15328.368883] LNet: Removed LNI 192.168.202.156@tcp [15329.601813] Key type .llcrypt unregistered [15329.604816] Key type ._llcrypt unregistered [15349.296803] Key type ._llcrypt registered [15349.302211] Key type .llcrypt registered [15349.469717] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [15351.017599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15351.291792] alg: No test for adler32 (adler32-zlib) [15352.486727] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [15352.882816] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [15354.602119] Key type lgssc registered [15356.065789] Lustre: Echo OBD driver; http://www.lustre.org/ [15360.414173] LNet: Added LNI 192.168.202.156@tcp100 [8/256/0/180] [15361.438513] LNet: Added LNI 192.168.202.156@tcp101 [8/256/0/180] [15362.402671] LNet: Added LNI 192.168.202.156@tcp102 [8/256/0/180] [15363.208523] LNet: Added LNI 192.168.202.156@tcp103 [8/256/0/180] [15364.087285] LNet: Added LNI 192.168.202.156@tcp104 [8/256/0/180] [15365.198952] LNet: Added LNI 192.168.202.156@tcp105 [8/256/0/180] [15366.130218] LNet: Added LNI 192.168.202.156@tcp106 [8/256/0/180] [15367.193062] LNet: Added LNI 192.168.202.156@tcp107 [8/256/0/180] [15368.090808] LNet: Added LNI 192.168.202.156@tcp108 [8/256/0/180] [15369.149604] LNet: Added LNI 192.168.202.156@tcp109 [8/256/0/180] [15370.112398] LNet: Added LNI 192.168.202.156@tcp110 [8/256/0/180] [15371.177285] LNet: Added LNI 192.168.202.156@tcp111 [8/256/0/180] [15372.317878] LNet: Added LNI 192.168.202.156@tcp112 [8/256/0/180] [15373.239603] LNet: Added LNI 192.168.202.156@tcp113 [8/256/0/180] [15374.041756] LNet: Added LNI 192.168.202.156@tcp114 [8/256/0/180] [15375.064687] LNet: Added LNI 192.168.202.156@tcp115 [8/256/0/180] [15376.030112] LNet: Added LNI 192.168.202.156@tcp116 [8/256/0/180] [15377.347745] LNet: Added LNI 192.168.202.156@tcp117 [8/256/0/180] [15378.221369] LNet: Added LNI 192.168.202.156@tcp118 [8/256/0/180] [15379.163463] LNet: Added LNI 192.168.202.156@tcp119 [8/256/0/180] [15380.092048] LNet: Added LNI 192.168.202.156@tcp120 [8/256/0/180] [15380.989469] LNet: Added LNI 192.168.202.156@tcp121 [8/256/0/180] [15382.052212] LNet: Added LNI 192.168.202.156@tcp122 [8/256/0/180] [15383.316383] LNet: Added LNI 192.168.202.156@tcp123 [8/256/0/180] [15384.242660] LNet: Added LNI 192.168.202.156@tcp124 [8/256/0/180] [15385.162980] LNet: Added LNI 192.168.202.156@tcp125 [8/256/0/180] [15386.310655] LNet: Added LNI 192.168.202.156@tcp126 [8/256/0/180] [15387.244271] LNet: Added LNI 192.168.202.156@tcp127 [8/256/0/180] [15388.314398] LNet: Added LNI 192.168.202.156@tcp128 [8/256/0/180] [15389.529764] LNet: Added LNI 192.168.202.156@tcp129 [8/256/0/180] [15390.782185] LNet: Added LNI 192.168.202.156@tcp130 [8/256/0/180] [15391.748278] LNet: Added LNI 192.168.202.156@tcp131 [8/256/0/180] [15393.038344] LNet: Added LNI 192.168.202.156@tcp132 [8/256/0/180] [15393.794464] LNet: Added LNI 192.168.202.156@tcp133 [8/256/0/180] [15394.907429] LNet: Added LNI 192.168.202.156@tcp134 [8/256/0/180] [15395.743045] LNet: Added LNI 192.168.202.156@tcp135 [8/256/0/180] [15403.355309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15403.449850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15404.675256] Lustre: 464445:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [15404.740575] LNetError: 464465:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [15404.762914] LNetError: Protocol error connecting to 192.168.202.156@tcp113 at host 192.168.202.156:988: is it running a compatible version of Lustre? [15404.774620] LNetError: Skipped 1 previous similar message [15404.989708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15405.042188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15405.163345] Lustre: lustre-MDT0000: new disk, initializing [15405.265899] Lustre: 464500:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [15405.281795] Lustre: 464500:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15405.322704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15405.422057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15405.794363] LNetError: 464468:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [15405.802737] LNetError: 464468:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15405.820414] LNetError: Protocol error connecting to 192.168.202.156@tcp100 at host 192.168.202.156:988: is it running a compatible version of Lustre? [15405.842176] LNetError: Skipped 1 previous similar message [15412.058438] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [15424.295623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15424.499817] Lustre: 464461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15424.589773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15424.593111] Lustre: Skipped 1 previous similar message [15424.691367] Lustre: lustre-MDT0001: new disk, initializing [15424.737834] Lustre: 465262:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [15424.753582] Lustre: 465262:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15424.968504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15425.048144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15425.082647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15429.739743] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [15434.788501] Lustre: 465694:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [15434.797525] Lustre: 465694:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15438.887841] LNetError: 464465:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [15438.914634] LNetError: Protocol error connecting to 192.168.202.156@tcp120 at host 192.168.202.156:988: is it running a compatible version of Lustre? [15439.730437] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15442.981424] LNetError: 464467:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [15443.005702] LNetError: Protocol error connecting to 192.168.202.156@tcp113 at host 192.168.202.156:988: is it running a compatible version of Lustre? [15446.959961] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15451.170357] LNetError: 461900:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [15451.194438] LNetError: 461900:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15451.212390] LNetError: Protocol error connecting to 192.168.202.156@tcp120 at host 192.168.202.156:988: is it running a compatible version of Lustre? [15451.225108] LNetError: Skipped 1 previous similar message [15454.185971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15454.479851] Lustre: lustre-OST0000: new disk, initializing [15454.488361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15454.499030] Lustre: 466435:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15454.656757] Lustre: 466440:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [15454.663839] Lustre: 466440:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15454.751601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15460.088823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15460.116384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15460.519074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15463.221693] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [15463.484132] LNetError: 464466:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [15463.502749] LNetError: 464466:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15463.514638] LNetError: Protocol error connecting to 192.168.202.156@tcp120 at host 192.168.202.156:988: is it running a compatible version of Lustre? [15463.530809] LNetError: Skipped 2 previous similar messages [15474.157837] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15480.865709] LNetError: 464467:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [15480.887562] LNetError: 464467:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15480.900801] LNetError: Protocol error connecting to 192.168.202.156@tcp113 at host 192.168.202.156:988: is it running a compatible version of Lustre? [15480.917896] LNetError: Skipped 3 previous similar messages [15485.923535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15485.924220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15485.938213] LustreError: Skipped 1 previous similar message [15485.939040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15485.961605] Lustre: Skipped 1 previous similar message [15489.927243] Lustre: server umount lustre-OST0000 complete [15496.164500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15496.167114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15496.182268] Lustre: Skipped 1 previous similar message [15496.233320] Lustre: Skipped 2 previous similar messages [15500.933180] Lustre: server umount lustre-MDT0000 complete [15505.197625] LustreError: 464455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788768097 with bad export cookie 10749762927168187087 [15505.209289] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15505.213842] LustreError: 464455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15505.733665] Lustre: server umount lustre-MDT0001 complete [15518.079611] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [15521.274731] Key type lgssc unregistered [15521.739393] LNet: 468361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15521.748720] LNetError: 468361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15522.784215] LNet: Removed LNI 192.168.202.156@tcp [15522.790145] LNetError: 468361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15523.810059] LNet: Removed LNI 192.168.202.156@tcp100 [15523.828350] LNetError: 468361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15523.856021] LNet: Removed LNI 192.168.202.156@tcp101 [15524.897381] LNet: Removed LNI 192.168.202.156@tcp102 [15525.920176] LNet: Removed LNI 192.168.202.156@tcp103 [15525.925464] LNetError: 468361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15525.944039] LNetError: 468361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15525.974262] LNet: Removed LNI 192.168.202.156@tcp104 [15527.008631] LNet: Removed LNI 192.168.202.156@tcp105 [15528.032163] LNet: Removed LNI 192.168.202.156@tcp106 [15529.056710] LNet: Removed LNI 192.168.202.156@tcp107 [15530.081618] LNet: Removed LNI 192.168.202.156@tcp108 [15530.106217] LNetError: 468361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15530.116696] LNetError: 468361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15530.123067] LNet: Removed LNI 192.168.202.156@tcp109 [15531.168181] LNet: Removed LNI 192.168.202.156@tcp110 [15532.192170] LNet: Removed LNI 192.168.202.156@tcp111 [15533.216168] LNet: Removed LNI 192.168.202.156@tcp112 [15534.241476] LNet: Removed LNI 192.168.202.156@tcp113 [15535.264192] LNet: Removed LNI 192.168.202.156@tcp114 [15536.288156] LNet: Removed LNI 192.168.202.156@tcp115 [15537.312158] LNet: Removed LNI 192.168.202.156@tcp116 [15538.400649] LNet: Removed LNI 192.168.202.156@tcp117 [15538.406230] LNetError: 468361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15538.421086] LNetError: 468361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15538.437617] LNet: Removed LNI 192.168.202.156@tcp118 [15539.488984] LNet: Removed LNI 192.168.202.156@tcp119 [15540.512882] LNet: Removed LNI 192.168.202.156@tcp120 [15540.519577] LNet: Removed LNI 192.168.202.156@tcp121 [15540.526406] LNet: Removed LNI 192.168.202.156@tcp122 [15540.529156] LNet: Removed LNI 192.168.202.156@tcp123 [15540.532836] LNet: Removed LNI 192.168.202.156@tcp124 [15540.535706] LNet: Removed LNI 192.168.202.156@tcp125 [15540.538768] LNet: Removed LNI 192.168.202.156@tcp126 [15540.547264] LNet: Removed LNI 192.168.202.156@tcp127 [15540.551886] LNet: Removed LNI 192.168.202.156@tcp128 [15540.558802] LNet: Removed LNI 192.168.202.156@tcp129 [15540.567605] LNet: Removed LNI 192.168.202.156@tcp130 [15540.571032] LNet: Removed LNI 192.168.202.156@tcp131 [15540.574844] LNet: Removed LNI 192.168.202.156@tcp132 [15540.579189] LNet: Removed LNI 192.168.202.156@tcp133 [15540.588617] LNet: Removed LNI 192.168.202.156@tcp134 [15540.607405] LNet: Removed LNI 192.168.202.156@tcp135 [15541.518261] Key type .llcrypt unregistered [15541.519895] Key type ._llcrypt unregistered [15571.286542] Key type ._llcrypt registered [15571.290569] Key type .llcrypt registered [15571.442868] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [15590.795243] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [15592.320971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15592.372531] alg: No test for adler32 (adler32-zlib) [15593.537560] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [15593.858470] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [15595.571326] Key type lgssc registered [15597.290919] Lustre: Echo OBD driver; http://www.lustre.org/ [15634.759664] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:03:44 (1788768224) [15656.383621] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [15659.852719] Key type lgssc unregistered [15660.189798] LNet: 472448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15660.204850] LNetError: 472448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15660.236504] LNet: Removed LNI 192.168.202.156@tcp [15661.364738] Key type .llcrypt unregistered [15661.366603] Key type ._llcrypt unregistered [15680.670752] Key type ._llcrypt registered [15680.674173] Key type .llcrypt registered [15680.791100] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [15682.133870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15682.185716] alg: No test for adler32 (adler32-zlib) [15683.389235] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [15683.786753] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [15685.632350] Key type lgssc registered [15687.118560] Lustre: Echo OBD driver; http://www.lustre.org/ [15699.299132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15699.337580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15700.750760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15700.819936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15700.880560] Lustre: lustre-MDT0000: new disk, initializing [15700.944470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15700.960605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15707.134665] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [15720.672906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15720.931195] Lustre: 473662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15721.025959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15721.029211] Lustre: Skipped 1 previous similar message [15721.144478] Lustre: lustre-MDT0001: new disk, initializing [15721.298894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15721.389633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15721.414496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15727.132460] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [15738.908799] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15745.686293] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15752.595902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15753.003534] Lustre: lustre-OST0000: new disk, initializing [15753.009461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15753.016451] Lustre: 475633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15753.182375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15758.966623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15758.986158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15759.181438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15761.187805] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [15771.827477] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15776.488788] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing lnet_if_list [15783.496976] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15783.522527] LNet: Added LNI 192.168.202.156@tcp10 [8/256/0/180] [15791.225375] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15791.247826] LNet: Added LNI 192.168.202.156@tcp11 [8/256/0/180] [15795.313682] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15795.342491] LNet: Added LNI 192.168.202.156@tcp12 [8/256/0/180] [15801.452549] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15801.488280] LNet: Added LNI 192.168.202.156@tcp13 [8/256/0/180] [15808.288645] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15808.338360] LNet: Added LNI 192.168.202.156@tcp14 [8/256/0/180] [15817.344602] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15817.416239] LNet: Added LNI 192.168.202.156@tcp15 [8/256/0/180] [15824.203870] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15824.248663] LNet: Added LNI 192.168.202.156@tcp16 [8/256/0/180] [15829.263847] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15829.317412] LNet: Added LNI 192.168.202.156@tcp17 [8/256/0/180] [15835.600411] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15835.646396] LNet: Added LNI 192.168.202.156@tcp18 [8/256/0/180] [15841.709148] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15841.791163] LNet: Added LNI 192.168.202.156@tcp19 [8/256/0/180] [15848.169206] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15848.217635] LNet: Added LNI 192.168.202.156@tcp20 [8/256/0/180] [15854.305951] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15854.345034] LNet: Added LNI 192.168.202.156@tcp21 [8/256/0/180] [15859.998645] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15860.020972] LNet: Added LNI 192.168.202.156@tcp22 [8/256/0/180] [15866.565791] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15866.610170] LNet: Added LNI 192.168.202.156@tcp23 [8/256/0/180] [15873.231880] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15873.272796] LNet: Added LNI 192.168.202.156@tcp24 [8/256/0/180] [15880.459729] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15880.491356] LNet: Added LNI 192.168.202.156@tcp25 [8/256/0/180] [15887.289669] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15887.332679] LNet: Added LNI 192.168.202.156@tcp26 [8/256/0/180] [15894.288679] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15894.323710] LNet: Added LNI 192.168.202.156@tcp27 [8/256/0/180] [15901.949718] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15901.977479] LNet: Added LNI 192.168.202.156@tcp28 [8/256/0/180] [15908.550722] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15908.580135] LNet: Added LNI 192.168.202.156@tcp29 [8/256/0/180] [15918.767938] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15918.824389] LNet: Added LNI 192.168.202.156@tcp30 [8/256/0/180] [15926.285996] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15926.306200] LNet: Added LNI 192.168.202.156@tcp31 [8/256/0/180] [15934.471608] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15934.517712] LNet: Added LNI 192.168.202.156@tcp32 [8/256/0/180] [15943.343489] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15943.407658] LNet: Added LNI 192.168.202.156@tcp33 [8/256/0/180] [15952.403759] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15952.471818] LNet: Added LNI 192.168.202.156@tcp34 [8/256/0/180] [15962.589422] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15962.640732] LNet: Added LNI 192.168.202.156@tcp35 [8/256/0/180] [15971.125565] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15971.149393] LNet: Added LNI 192.168.202.156@tcp36 [8/256/0/180] [15980.843446] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15980.898558] LNet: Added LNI 192.168.202.156@tcp37 [8/256/0/180] [15988.340930] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15988.362385] LNet: Added LNI 192.168.202.156@tcp38 [8/256/0/180] [15995.397811] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15995.421294] LNet: Added LNI 192.168.202.156@tcp39 [8/256/0/180] [16002.869555] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16002.896906] LNet: Added LNI 192.168.202.156@tcp40 [8/256/0/180] [16015.088813] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16015.141124] LNet: Added LNI 192.168.202.156@tcp41 [8/256/0/180] [16026.899494] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16026.940510] LNet: Added LNI 192.168.202.156@tcp42 [8/256/0/180] [16036.549717] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16036.604644] LNet: Added LNI 192.168.202.156@tcp43 [8/256/0/180] [16047.277489] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16047.322356] LNet: Added LNI 192.168.202.156@tcp44 [8/256/0/180] [16058.332394] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16058.353823] LNet: Added LNI 192.168.202.156@tcp45 [8/256/0/180] [16068.552315] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16068.637824] LNet: Added LNI 192.168.202.156@tcp46 [8/256/0/180] [16078.692469] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16078.720416] LNet: Added LNI 192.168.202.156@tcp47 [8/256/0/180] [16089.137519] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16089.169076] LNet: Added LNI 192.168.202.156@tcp48 [8/256/0/180] [16099.349867] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16099.402259] LNet: Added LNI 192.168.202.156@tcp49 [8/256/0/180] [16109.998936] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16110.051099] LNet: Added LNI 192.168.202.156@tcp50 [8/256/0/180] [16131.041554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16131.051110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16131.071788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16133.091421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16133.093831] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16133.107652] Lustre: Skipped 1 previous similar message [16133.131617] Lustre: Skipped 1 previous similar message [16135.999358] Lustre: server umount lustre-MDT0000 complete [16139.674312] LustreError: 473655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788768732 with bad export cookie 475251598903333619 [16139.688053] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16139.693950] LustreError: 473655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16139.940497] Lustre: server umount lustre-MDT0001 complete [16159.716448] Lustre: 472932:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788768736/real 1788768736] req@000000005aee25e1 x1875659483967104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788768752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16159.759103] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16160.365280] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [16160.736398] Lustre: 472933:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788768737/real 1788768737] req@000000006df4f914 x1875659483967232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788768753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16164.832805] Lustre: 472935:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788768741/real 1788768741] req@000000007bbbb7b8 x1875659483967360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788768757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16170.987067] Lustre: 472933:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788768747/real 1788768747] req@000000006453e42c x1875659483967744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788768763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16171.031245] Lustre: 472933:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16172.116765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16176.160335] Lustre: 472933:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788768752/real 1788768752] req@00000000b8cc230d x1875659483967872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788768768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16184.873888] LustreError: 475626:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16184.902580] LustreError: 475626:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [16184.990919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16189.093328] LustreError: 475627:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16189.104265] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16190.102909] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16194.532699] LustreError: 475626:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16199.312990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16199.785708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16204.941714] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16205.856942] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16214.436799] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16221.279261] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16245.729827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16245.753329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16245.771306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16245.789176] Lustre: Skipped 1 previous similar message [16250.468983] Lustre: server umount lustre-OST0000 complete [16255.969930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16255.979596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16256.002319] Lustre: Skipped 1 previous similar message [16256.017546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16256.026168] Lustre: Skipped 2 previous similar messages [16261.607772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16261.624807] Lustre: Skipped 3 previous similar messages [16269.280187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16269.624823] Lustre: server umount lustre-MDT0000 complete [16271.848209] LustreError: 501329:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16271.879132] LustreError: 501329:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [16273.370487] LustreError: 473656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788768866 with bad export cookie 475251598903336265 [16273.390744] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16273.397776] LustreError: 473656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16273.754658] Lustre: server umount lustre-MDT0001 complete [16285.070748] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [16287.495099] Key type lgssc unregistered [16287.812417] LNet: 502685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16287.820176] LNetError: 502685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16287.824563] LNet: Removed LNI 192.168.202.156@tcp [16287.830590] LNet: Removed LNI 192.168.202.156@tcp10 [16287.833756] LNet: Removed LNI 192.168.202.156@tcp11 [16287.837972] LNet: Removed LNI 192.168.202.156@tcp12 [16287.840194] LNet: Removed LNI 192.168.202.156@tcp13 [16287.842844] LNet: Removed LNI 192.168.202.156@tcp14 [16287.847170] LNet: Removed LNI 192.168.202.156@tcp15 [16287.849030] LNet: Removed LNI 192.168.202.156@tcp16 [16287.859418] LNet: Removed LNI 192.168.202.156@tcp17 [16287.863752] LNet: Removed LNI 192.168.202.156@tcp18 [16287.866332] LNet: Removed LNI 192.168.202.156@tcp19 [16287.870653] LNet: Removed LNI 192.168.202.156@tcp20 [16287.875062] LNet: Removed LNI 192.168.202.156@tcp21 [16287.879832] LNet: Removed LNI 192.168.202.156@tcp22 [16287.883897] LNet: Removed LNI 192.168.202.156@tcp23 [16287.887386] LNet: Removed LNI 192.168.202.156@tcp24 [16287.890276] LNet: Removed LNI 192.168.202.156@tcp25 [16287.894115] LNet: Removed LNI 192.168.202.156@tcp26 [16287.899043] LNet: Removed LNI 192.168.202.156@tcp27 [16287.901299] LNet: Removed LNI 192.168.202.156@tcp28 [16287.906738] LNet: Removed LNI 192.168.202.156@tcp29 [16287.912382] LNet: Removed LNI 192.168.202.156@tcp30 [16287.919829] LNet: Removed LNI 192.168.202.156@tcp31 [16287.933289] LNet: Removed LNI 192.168.202.156@tcp32 [16287.936768] LNet: Removed LNI 192.168.202.156@tcp33 [16287.941737] LNet: Removed LNI 192.168.202.156@tcp34 [16287.943413] LNet: Removed LNI 192.168.202.156@tcp35 [16287.953302] LNet: Removed LNI 192.168.202.156@tcp36 [16287.957306] LNet: Removed LNI 192.168.202.156@tcp37 [16287.961289] LNet: Removed LNI 192.168.202.156@tcp38 [16287.969362] LNet: Removed LNI 192.168.202.156@tcp39 [16287.973482] LNet: Removed LNI 192.168.202.156@tcp40 [16287.975975] LNet: Removed LNI 192.168.202.156@tcp41 [16287.979981] LNet: Removed LNI 192.168.202.156@tcp42 [16287.984348] LNet: Removed LNI 192.168.202.156@tcp43 [16287.987485] LNet: Removed LNI 192.168.202.156@tcp44 [16287.991195] LNet: Removed LNI 192.168.202.156@tcp45 [16287.994860] LNet: Removed LNI 192.168.202.156@tcp46 [16287.997467] LNet: Removed LNI 192.168.202.156@tcp47 [16288.001510] LNet: Removed LNI 192.168.202.156@tcp48 [16288.005204] LNet: Removed LNI 192.168.202.156@tcp49 [16288.017737] LNet: Removed LNI 192.168.202.156@tcp50 [16288.684799] Key type .llcrypt unregistered [16288.688235] Key type ._llcrypt unregistered [16311.273147] Key type ._llcrypt registered [16311.274936] Key type .llcrypt registered [16311.426124] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [16329.197083] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [16330.843377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16330.882222] alg: No test for adler32 (adler32-zlib) [16331.939371] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [16332.423231] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [16334.281145] Key type lgssc registered [16335.965629] Lustre: Echo OBD driver; http://www.lustre.org/ [16374.099759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16374.115413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16375.508677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16375.558697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16375.740923] Lustre: lustre-MDT0000: new disk, initializing [16375.874796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16375.909046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16381.901936] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16395.751807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16396.049931] Lustre: 505884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16396.194260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16396.207859] Lustre: Skipped 1 previous similar message [16396.400565] Lustre: lustre-MDT0001: new disk, initializing [16396.620512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16396.697132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16396.713408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16403.899993] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16415.493726] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16422.438888] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16431.964365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16432.241695] Lustre: *** cfs_fail_loc=90f, val=0*** [16432.405678] Lustre: lustre-OST0000: new disk, initializing [16432.413159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16432.418717] Lustre: 507901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16432.641049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16439.873469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16439.881409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16440.018595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16440.422272] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16451.769628] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16465.895886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16465.911688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16465.919116] Lustre: Skipped 1 previous similar message [16465.973092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16467.535824] Lustre: server umount lustre-OST0000 complete [16473.569993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16473.581289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16473.597088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16473.602662] Lustre: Skipped 1 previous similar message [16476.142988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16476.149310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16476.156503] Lustre: Skipped 1 previous similar message [16481.249918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16481.255240] Lustre: Skipped 1 previous similar message [16485.346493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16485.359813] Lustre: Skipped 1 previous similar message [16486.368259] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16486.478624] Lustre: server umount lustre-MDT0000 complete [16490.654817] LustreError: 507900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788769083 with bad export cookie 8473575040223787522 [16490.656627] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16490.669242] LustreError: 507900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16491.244591] Lustre: server umount lustre-MDT0001 complete [16505.567444] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [16508.898397] Key type lgssc unregistered [16509.210285] LNet: 509810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16509.218201] LNetError: 509810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16509.239627] LNet: Removed LNI 192.168.202.156@tcp [16510.198072] Key type .llcrypt unregistered [16510.199936] Key type ._llcrypt unregistered [16532.552404] Key type ._llcrypt registered [16532.560881] Key type .llcrypt registered [16532.783368] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [16535.060595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16535.451376] alg: No test for adler32 (adler32-zlib) [16537.098931] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [16537.647104] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [16539.488221] Key type lgssc registered [16541.906764] Lustre: Echo OBD driver; http://www.lustre.org/ [16554.435521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16554.452690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16556.055647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16560.545935] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16566.263852] LustreError: 511076:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16566.283864] LustreError: 511076:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [16569.129645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16569.502042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16573.997364] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16582.993188] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16589.201960] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16597.902995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16598.279250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16604.609547] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16613.411782] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16625.341897] loop0: detected capacity change from 0 to 204800000 [16625.583696] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16625.593315] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16625.605939] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16632.142924] loop0: detected capacity change from 0 to 204800000 [16632.394299] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16632.401759] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16632.422098] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16638.366807] loop0: detected capacity change from 0 to 204800000 [16644.254583] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16644.433390] Lustre: fs15246-OST0000: new disk, initializing [16644.437232] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16644.442846] Lustre: 514669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16644.489623] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16651.457857] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16662.466994] loop1: detected capacity change from 0 to 204800000 [16667.832542] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16667.953346] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16668.011804] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16668.088974] Lustre: fs15246-MDT0000: new disk, initializing [16668.222966] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16668.250298] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16668.362861] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16669.413248] Lustre: 515789:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001b7eccec x1875660379308160/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788769278 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16673.195718] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16678.504229] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16678.532346] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16678.560271] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16696.297907] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16696.321590] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16699.974689] Lustre: server umount fs15246-MDT0000 complete [16701.424832] LustreError: 516650:0:(ldlm_lib.c:1202:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16706.543153] LustreError: 512112:0:(ldlm_lib.c:1202:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16709.940307] Lustre: server umount fs15246-OST0000 complete [16721.892670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16721.900107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16721.901545] Lustre: Skipped 1 previous similar message [16721.917197] Lustre: Skipped 1 previous similar message [16727.015600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16727.026653] Lustre: Skipped 1 previous similar message [16727.734341] Lustre: server umount lustre-OST0000 complete [16733.161446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16733.186291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16733.210485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16737.250055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16737.250227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16738.881373] Lustre: server umount lustre-MDT0000 complete [16742.369975] LustreError: 512112:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16742.391706] LustreError: 512112:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [16743.007383] LustreError: 511055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788769335 with bad export cookie 2428711282450450520 [16743.010774] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16743.019153] LustreError: 511055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16743.407887] Lustre: server umount lustre-MDT0001 complete [16756.662772] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [16760.571596] Key type lgssc unregistered [16760.941596] LNet: 518889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16760.958464] LNetError: 518889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16762.022954] LNet: Removed LNI 192.168.202.156@tcp [16762.966607] Key type .llcrypt unregistered [16762.968880] Key type ._llcrypt unregistered [16784.388478] Key type ._llcrypt registered [16784.391470] Key type .llcrypt registered [16784.520378] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [16785.952659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16786.075635] alg: No test for adler32 (adler32-zlib) [16787.382037] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [16787.707514] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [16789.442728] Key type lgssc registered [16790.778367] Lustre: Echo OBD driver; http://www.lustre.org/ [16833.638368] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [16845.601377] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [16883.263405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16883.277912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16884.552933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16884.579952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16884.651335] Lustre: lustre-MDT0000: new disk, initializing [16884.720164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16884.751375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16888.993494] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16900.785207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16900.953235] Lustre: 523354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16901.035263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16901.039191] Lustre: Skipped 1 previous similar message [16901.154252] Lustre: lustre-MDT0001: new disk, initializing [16901.302386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16901.344597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16901.358194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16906.591295] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16916.290800] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16922.568671] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16929.154066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16929.419240] Lustre: lustre-OST0000: new disk, initializing [16929.422827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16929.432429] Lustre: 525326:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16929.513356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16935.057813] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [16939.146027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16939.156179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16939.272495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16943.911902] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16949.574145] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16949.862130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16955.856591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16956.150807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16959.971581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16959.989451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16960.021295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16965.108259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16965.123619] Lustre: Skipped 2 previous similar messages [16970.215952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16970.224472] Lustre: Skipped 1 previous similar message [16972.257050] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16972.367712] Lustre: server umount lustre-OST0000 complete [16978.402174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16978.407707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16978.412847] Lustre: Skipped 1 previous similar message [16978.416358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16980.453974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16985.570343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16985.574799] Lustre: Skipped 3 previous similar messages [16991.200122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16991.561763] Lustre: server umount lustre-MDT0000 complete [16995.138556] LustreError: 523348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788769587 with bad export cookie 9744190676777986017 [16995.155162] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16995.584925] Lustre: server umount lustre-MDT0001 complete [17021.103662] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [17031.784598] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [17065.390253] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16738 sec ========= 04:27:35 (1788769655) [17067.325734] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:27:37 (1788769657) === [17071.137766] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:27:41 (1788769661) === [17110.380144] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [17113.141253] Key type lgssc unregistered [17113.380639] LNet: 531973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17113.388431] LNetError: 531973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17113.417833] LNet: Removed LNI 192.168.202.156@tcp [17114.266708] Key type .llcrypt unregistered [17114.267847] Key type ._llcrypt unregistered