[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507857928 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003345] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008766] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012063] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.023382] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028559] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029180] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031181] *** VALIDATE tmpfs *** [ 0.033429] *** VALIDATE proc *** [ 0.035133] *** VALIDATE cgroup *** [ 0.036014] *** VALIDATE cgroup2 *** [ 0.038204] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.042018] Speculative Store Bypass: Vulnerable [ 0.045598] debug: unmapping init [mem 0xffffffffbda59000-0xffffffffbda60fff] [ 0.047969] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048779] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049029] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052017] ... value mask: 0000ffffffffffff [ 0.053010] ... max period: 00007fffffffffff [ 0.054013] ... fixed-purpose events: 3 [ 0.054977] ... event mask: 000000070000000f [ 0.055371] rcu: Hierarchical SRCU implementation. [ 0.057568] smp: Bringing up secondary CPUs ... [ 0.058634] x86: Booting SMP configuration: [ 0.059034] .... node #0, CPUs: #1 #2 #3 [ 0.062513] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.119031] node 0 deferred pages initialised in 51ms [ 0.122297] devtmpfs: initialized [ 0.123456] x86/mm: Memory block size: 128MB [ 0.126951] gcov: version magic: 0x41383552 [ 0.129320] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.132120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.135351] pinctrl core: initialized pinctrl subsystem [ 0.136177] [ 0.136693] ************************************************************* [ 0.139013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.141011] ** ** [ 0.142012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145017] ** ** [ 0.147014] ** This means that this kernel is built to expose internal ** [ 0.149013] ** IOMMU data structures, which may compromise security on ** [ 0.151013] ** your system. ** [ 0.153013] ** ** [ 0.155029] ** If you see this message and you are not debugging the ** [ 0.158015] ** kernel, report this immediately to your vendor! ** [ 0.159012] ** ** [ 0.161015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164015] ************************************************************* [ 0.166713] NET: Registered protocol family 16 [ 0.168440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176036] cpuidle: using governor menu [ 0.177863] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180494] PCI: Using configuration type 1 for base access [ 0.183142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194144] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198113] cryptd: max_cpu_qlen set to 1000 [ 0.202427] ACPI: Added _OSI(Module Device) [ 0.205017] ACPI: Added _OSI(Processor Device) [ 0.208018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215157] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223553] ACPI: Interpreter enabled [ 0.226056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228015] ACPI: Using IOAPIC for interrupt routing [ 0.230139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.233438] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242805] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.246051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259367] acpiphp: Slot [2] registered [ 0.261128] acpiphp: Slot [5] registered [ 0.262109] acpiphp: Slot [6] registered [ 0.264116] acpiphp: Slot [7] registered [ 0.265094] acpiphp: Slot [8] registered [ 0.266115] acpiphp: Slot [9] registered [ 0.267109] acpiphp: Slot [10] registered [ 0.269118] acpiphp: Slot [3] registered [ 0.270103] acpiphp: Slot [4] registered [ 0.271089] acpiphp: Slot [11] registered [ 0.272096] acpiphp: Slot [12] registered [ 0.274097] acpiphp: Slot [13] registered [ 0.275152] acpiphp: Slot [14] registered [ 0.278142] acpiphp: Slot [15] registered [ 0.280157] acpiphp: Slot [16] registered [ 0.283160] acpiphp: Slot [17] registered [ 0.285154] acpiphp: Slot [18] registered [ 0.287175] acpiphp: Slot [19] registered [ 0.289136] acpiphp: Slot [20] registered [ 0.292120] acpiphp: Slot [21] registered [ 0.294110] acpiphp: Slot [22] registered [ 0.296091] acpiphp: Slot [23] registered [ 0.297103] acpiphp: Slot [24] registered [ 0.298120] acpiphp: Slot [25] registered [ 0.300094] acpiphp: Slot [26] registered [ 0.301098] acpiphp: Slot [27] registered [ 0.302123] acpiphp: Slot [28] registered [ 0.304088] acpiphp: Slot [29] registered [ 0.305130] acpiphp: Slot [30] registered [ 0.306105] acpiphp: Slot [31] registered [ 0.307061] PCI host bridge to bus 0000:00 [ 0.309019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.311033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.316023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320820] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.323181] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.337055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.340026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.344020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346633] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.350089] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.353050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.359016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384271] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.404020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.418023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.435897] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.443017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.450016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.478000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.486018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.490000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.515021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.524000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.531019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.539023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.560018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.568200] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.576019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.582015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.602016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.613870] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.622017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.631018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.648020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.659407] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.661370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.664415] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.666461] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.669387] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.675098] iommu: Default domain type: Passthrough [ 0.677412] SCSI subsystem initialized [ 0.679209] ACPI: bus type USB registered [ 0.681186] usbcore: registered new interface driver usbfs [ 0.682137] usbcore: registered new interface driver hub [ 0.685091] usbcore: registered new device driver usb [ 0.687191] pps_core: LinuxPPS API ver. 1 registered [ 0.689013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.692146] PTP clock support registered [ 0.694158] EDAC MC: Ver: 3.0.0 [ 0.696135] PCI: Using ACPI for IRQ routing [ 0.697562] NetLabel: Initializing [ 0.699016] NetLabel: domain hash size = 128 [ 0.700011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.702093] NetLabel: unlabeled traffic allowed by default [ 0.704126] vgaarb: loaded [ 0.706348] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.708017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.714126] clocksource: Switched to clocksource kvm-clock [ 0.820412] VFS: Disk quotas dquot_6.6.0 [ 0.822242] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.824891] *** VALIDATE ramfs *** [ 0.825978] *** VALIDATE hugetlbfs *** [ 0.827024] pnp: PnP ACPI init [ 0.828878] pnp: PnP ACPI: found 6 devices [ 0.847616] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.851086] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.853023] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.854904] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.857042] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.859294] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.862040] NET: Registered protocol family 2 [ 0.864212] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.868616] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.871802] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.876783] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.879920] TCP: Hash tables configured (established 65536 bind 65536) [ 0.882254] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.884765] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.887820] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.890750] NET: Registered protocol family 1 [ 0.893845] RPC: Registered named UNIX socket transport module. [ 0.895840] RPC: Registered udp transport module. [ 0.897173] RPC: Registered tcp transport module. [ 0.898622] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.900685] NET: Registered protocol family 44 [ 0.901982] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.903776] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.905583] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.908016] PCI: CLS 0 bytes, default 64 [ 0.909431] Unpacking initramfs... [ 2.318724] debug: unmapping init [mem 0xffff9ac1bcc54000-0xffff9ac1bffbffff] [ 2.322708] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.325133] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.328131] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.827775] Initialise system trusted keyrings [ 2.829514] Key type blacklist registered [ 2.831450] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.840685] zbud: loaded [ 2.843729] *** VALIDATE nfs *** [ 2.845035] *** VALIDATE nfs4 *** [ 2.846835] pstore: using deflate compression [ 2.850606] Platform Keyring initialized [ 2.955321] NET: Registered protocol family 38 [ 2.956624] Key type asymmetric registered [ 2.957661] Asymmetric key parser 'x509' registered [ 2.958839] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.960914] io scheduler mq-deadline registered [ 2.962710] io scheduler kyber registered [ 2.964518] io scheduler bfq registered [ 2.966139] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.969376] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.972395] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.975326] ACPI: Power Button [PWRF] [ 2.980683] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.986507] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.007974] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.017382] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.035051] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.062527] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.090032] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.093762] Non-volatile memory driver v1.3 [ 3.095046] Linux agpgart interface v0.103 [ 3.125407] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.128237] vda: detected capacity change from 0 to 76775424 [ 3.144153] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.147469] vdb: detected capacity change from 0 to 1073741824 [ 3.162843] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.165456] vdc: detected capacity change from 0 to 2621440000 [ 3.180440] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.182465] vdd: detected capacity change from 0 to 2621440000 [ 3.200672] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.202964] vde: detected capacity change from 0 to 4294967296 [ 3.217390] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.219432] vdf: detected capacity change from 0 to 4294967296 [ 3.229764] libphy: Fixed MDIO Bus: probed [ 3.234461] usbcore: registered new interface driver usbserial_generic [ 3.237061] usbserial: USB Serial support registered for generic [ 3.239473] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.244361] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.246425] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.249134] mousedev: PS/2 mouse device common for all mice [ 3.251830] rtc_cmos 00:05: RTC can wake from S4 [ 3.255577] rtc_cmos 00:05: registered as rtc0 [ 3.256497] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.257508] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.264421] intel_pstate: CPU model not supported [ 3.267077] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.271710] hid: raw HID events driver (C) Jiri Kosina [ 3.274328] usbcore: registered new interface driver usbhid [ 3.274393] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.276196] usbhid: USB HID core driver [ 3.280848] drop_monitor: Initializing network drop monitor service [ 3.283221] Initializing XFRM netlink socket [ 3.285091] NET: Registered protocol family 10 [ 3.288123] Segment Routing with IPv6 [ 3.289414] NET: Registered protocol family 17 [ 3.291366] mpls_gso: MPLS GSO support [ 3.296603] RAS: Correctable Errors collector initialized. [ 3.298825] AVX version of gcm_enc/dec engaged. [ 3.300444] AES CTR mode by8 optimization enabled [ 3.370958] sched_clock: Marking stable (3370937847, 0)->(4316601310, -945663463) [ 3.373986] registered taskstats version 1 [ 3.376557] Loading compiled-in X.509 certificates [ 3.378302] zswap: loaded using pool lzo/zbud [ 3.399901] Key type big_key registered [ 3.412443] Key type encrypted registered [ 3.413853] ima: No TPM chip found, activating TPM-bypass! [ 3.415707] ima: Allocated hash algorithm: sha1 [ 3.417487] ima: No architecture policies found [ 3.419262] evm: Initialising EVM extended attributes: [ 3.421290] evm: security.selinux [ 3.422507] evm: security.ima [ 3.423571] evm: security.capability [ 3.424756] evm: HMAC attrs: 0x1 [ 3.426859] rtc_cmos 00:05: setting system clock to 2026-09-09 06:33:25 UTC (1788935605) [ 3.433399] debug: unmapping init [mem 0xffffffffbea03000-0xffffffffbebfffff] [ 3.436833] debug: unmapping init [mem 0xffffffffbd782000-0xffffffffbda58fff] [ 3.449071] Write protecting the kernel read-only data: 28672k [ 3.452526] debug: unmapping init [mem 0xffffffffbbe03000-0xffffffffbbffffff] [ 3.455292] debug: unmapping init [mem 0xffffffffbc714000-0xffffffffbc7fffff] [ 3.489073] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.496954] systemd[1]: Detected virtualization kvm. [ 3.499548] systemd[1]: Detected architecture x86-64. [ 3.501864] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.529154] systemd[1]: No hostname configured. [ 3.530712] systemd[1]: Set hostname to . [ 3.533276] random: systemd: uninitialized urandom read (16 bytes read) [ 3.535675] systemd[1]: Initializing machine ID from random generator. [ 3.663836] random: systemd: uninitialized urandom read (16 bytes read) [ 3.666611] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.671323] random: systemd: uninitialized urandom read (16 bytes read) [ 3.673038] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.676774] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.788068] device-mapper: uevent: version 1.0.3 [ 4.792984] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.798134] virtio_net virtio0 ens2: renamed from eth0 [ 6.948634] random: fast init done [ 7.156315] scsi host0: ata_piix [ 7.169273] scsi host1: ata_piix [ 7.173793] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.180227] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.956052] random: crng init done [ 11.959417] random: 7 urandom warning(s) missed due to ratelimiting [ 14.508728] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.619681] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.887392] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.881623] SELinux: Disabled at runtime. [ 21.078508] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.101130] systemd[1]: Detected virtualization kvm. [ 21.109228] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.990344] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.996044] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.004223] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.010451] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.017439] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.024701] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.037385] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ 23.535943] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.162513] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.396580] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.438160] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.022806] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.051302] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 30.937847] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.923659] NFS: Registering the id_resolver key type [ 31.927260] Key type id_resolver registered [ 31.930733] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg449-server login: [ 94.438259] libcfs: loading out-of-tree module taints kernel. [ 94.498068] Key type ._llcrypt registered [ 94.501670] Key type .llcrypt registered [ 94.703340] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 115.208165] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 117.110830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.123628] alg: No test for adler32 (adler32-zlib) [ 118.896550] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 119.610201] hrtimer: interrupt took 5192144 ns [ 120.053234] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 121.952261] Key type lgssc registered [ 123.867878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 141.162560] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.004823] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 200.929548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 200.996184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.532491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.589425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.716773] Lustre: lustre-MDT0000: new disk, initializing [ 202.811494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.838145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.614820] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.778113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.914231] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.967511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 223.979121] Lustre: Skipped 1 previous similar message [ 224.173989] Lustre: lustre-MDT0001: new disk, initializing [ 224.263700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.336868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 224.346301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 229.720143] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.823211] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 245.097368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.376446] Lustre: lustre-OST0000: new disk, initializing [ 245.380588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.389872] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.452498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.224711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 248.231544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 248.327521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.446383] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.986215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.161440] Lustre: lustre-OST0001: new disk, initializing [ 268.164875] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 268.170288] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.237787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 275.004717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.014246] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.106840] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 275.848525] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 288.516063] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 299.669474] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 306.799869] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing check_logdir /tmp/testlogs/ [ 312.378181] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing yml_node [ 316.296993] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 319.177590] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 322.147185] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 323.774465] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 02:38:45 EDT 2026 [ 342.405428] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 344.116691] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.907696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.912496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.923381] Lustre: Skipped 3 previous similar messages [ 359.930507] Lustre: Skipped 3 previous similar messages [ 365.030765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.037958] Lustre: Skipped 3 previous similar messages [ 365.446488] Lustre: server umount lustre-MDT0000 complete [ 369.219801] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935971 with bad export cookie 842993639836882307 [ 369.226666] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.240820] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 370.145247] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.148989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.154729] Lustre: Skipped 1 previous similar message [ 374.182893] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.188444] Lustre: Skipped 1 previous similar message [ 375.750234] Lustre: server umount lustre-MDT0001 complete [ 379.339089] Lustre: server umount lustre-OST0000 complete [ 383.943215] Lustre: server umount lustre-OST0001 complete [ 391.621375] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 402.697206] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 437.046990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.253983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 437.279460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 437.353497] Lustre: lustre-MDT0000: new disk, initializing [ 437.430681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.445332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 442.329885] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 452.922574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.083966] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 453.174872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 453.184325] Lustre: Skipped 1 previous similar message [ 453.283750] Lustre: lustre-MDT0001: new disk, initializing [ 453.411570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 453.501040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 453.508918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 458.387559] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 468.471610] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 474.714195] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 481.437502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 481.735310] Lustre: lustre-OST0000: new disk, initializing [ 481.739454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 481.745303] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 481.862507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.482184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 483.494038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 483.658445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 488.491723] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 498.095851] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 502.020928] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 502.269503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 507.141840] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 507.444486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 509.411256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.414144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 509.444127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.448677] Lustre: Skipped 2 previous similar messages [ 515.430288] Lustre: server umount lustre-OST0000 complete [ 519.651382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.655679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.671946] Lustre: Skipped 2 previous similar messages [ 519.701183] Lustre: Skipped 3 previous similar messages [ 525.698756] Lustre: server umount lustre-MDT0000 complete [ 529.892199] LustreError: 15085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.910627] LustreError: 15085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 530.364537] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936132 with bad export cookie 842993639836883840 [ 530.369373] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.376906] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 530.873591] Lustre: server umount lustre-MDT0001 complete [ 535.436482] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:42:16 (1788936136) [ 538.449206] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 540.507052] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:42:21 (1788936141) [ 547.817294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.333748] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.414529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.444924] LustreError: 19457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.930383] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 558.576811] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.684607] LustreError: 19457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.925290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.279852] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 579.106552] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 585.203126] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 593.224449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 600.231190] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 608.429994] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 619.502777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 619.514967] LustreError: Skipped 1 previous similar message [ 619.518716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.530349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.540022] Lustre: Skipped 3 previous similar messages [ 625.291649] Lustre: server umount lustre-OST0000 complete [ 629.740229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.758246] Lustre: Skipped 1 previous similar message [ 635.633047] Lustre: server umount lustre-MDT0000 complete [ 639.512783] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936241 with bad export cookie 842993639836884715 [ 639.518796] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.525277] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 640.123668] Lustre: server umount lustre-MDT0001 complete [ 653.105527] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 656.443382] Key type lgssc unregistered [ 656.744712] LNet: 22988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.756846] LNetError: 22988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.834267] LNet: Removed LNI 192.168.204.149@tcp [ 658.899968] Key type .llcrypt unregistered [ 658.906730] Key type ._llcrypt unregistered [ 682.386597] Key type ._llcrypt registered [ 682.388645] Key type .llcrypt registered [ 682.489217] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 683.780861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.797183] alg: No test for adler32 (adler32-zlib) [ 684.955800] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 685.320111] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 687.128694] Key type lgssc registered [ 688.642687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 698.305133] loop: module loaded [ 698.431339] loop0: detected capacity change from 0 to 204800000 [ 698.704853] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.731326] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.137494] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.940367] loop0: detected capacity change from 0 to 204800000 [ 707.266901] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.292083] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.322514] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.937109] loop0: detected capacity change from 0 to 204800000 [ 718.882330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 718.893165] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.196774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 720.247590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 720.371247] Lustre: lustre-MDT0000: new disk, initializing [ 720.501117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.539782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 727.889177] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 737.974712] loop1: detected capacity change from 0 to 204800000 [ 744.469392] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 744.828496] Lustre: lustre-OST07c6: new disk, initializing [ 744.837481] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 744.844145] Lustre: Skipped 1 previous similar message [ 744.851931] Lustre: 26173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 744.916899] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 751.792862] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 754.266902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 754.280985] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 754.312950] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 757.460867] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 757.471713] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 757.478873] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 757.487726] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 757.497261] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 757.507176] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 772.638941] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 772.645664] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.653550] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.662758] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 772.669313] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 772.676029] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.664434] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 777.674388] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 777.681499] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 777.687375] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.694471] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.708345] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.725571] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 777.740348] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.756199] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.771421] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.782585] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.795108] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.054989] Lustre: 25159:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 781.072475] Lustre: 25159:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 781.078884] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 781.097606] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.107150] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 781.115642] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.125982] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 781.131941] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.137641] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 781.141803] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.146154] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 781.150560] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.187956] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 786.200249] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 786.210913] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 786.222206] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.228655] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.236472] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.242651] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 786.252735] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.258905] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 786.268304] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.280242] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.286959] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.284855] Lustre: 25159:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 797.293868] Lustre: 25159:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 797.300590] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 797.304741] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 797.317408] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.326883] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 797.332104] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 797.336455] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 797.341274] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 797.346240] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 797.354681] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 797.362399] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 851.426183] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.439360] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 851.441095] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 852.571219] Lustre: server umount lustre-OST07c6 complete [ 862.835058] Lustre: server umount lustre-MDT0000 complete [ 880.061847] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 883.392688] Key type lgssc unregistered [ 883.839073] LNet: 28727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.852786] LNetError: 28727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.880972] LNet: Removed LNI 192.168.204.149@tcp [ 885.288373] Key type .llcrypt unregistered [ 885.294498] Key type ._llcrypt unregistered [ 906.567718] Key type ._llcrypt registered [ 906.570059] Key type .llcrypt registered [ 906.656882] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 907.940603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.956066] alg: No test for adler32 (adler32-zlib) [ 909.078619] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 909.308151] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 911.040695] Key type lgssc registered [ 912.536710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 921.434731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 921.455846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.138056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.507382] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 933.347617] LustreError: 29995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 933.382193] LustreError: 29995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 938.475894] LustreError: 29996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.857587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.430309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 945.050867] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 953.560330] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 959.700807] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 967.304802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 967.851872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.883847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 974.337136] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 981.889220] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 999.905773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.917224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.936545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1002.343664] Lustre: server umount lustre-OST0000 complete [ 1010.154281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.167615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1010.191957] Lustre: Skipped 2 previous similar messages [ 1010.201877] Lustre: Skipped 2 previous similar messages [ 1013.332254] Lustre: server umount lustre-MDT0000 complete [ 1015.266249] LustreError: 31105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.301365] LustreError: 31105:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1017.332168] LustreError: 29977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936619 with bad export cookie 15179615745135234332 [ 1017.333840] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1017.356799] LustreError: 29977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1017.646912] Lustre: server umount lustre-MDT0001 complete [ 1028.581708] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 1031.355458] Key type lgssc unregistered [ 1031.696966] LNet: 33733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.701217] LNetError: 33733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.743292] LNet: Removed LNI 192.168.204.149@tcp [ 1033.799825] Key type .llcrypt unregistered [ 1033.801400] Key type ._llcrypt unregistered [ 1054.574043] Key type ._llcrypt registered [ 1054.577389] Key type .llcrypt registered [ 1054.718983] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 1056.081968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.140362] alg: No test for adler32 (adler32-zlib) [ 1057.454982] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 1057.722712] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 1059.449416] Key type lgssc registered [ 1060.702924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1070.197001] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1070.228134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.046482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.438320] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1082.339946] LustreError: 35000:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.348506] LustreError: 35000:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1084.933223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.417326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1090.223671] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1097.902392] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1104.534704] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1112.299600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1112.728941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.888009] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1119.383941] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1127.094432] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1141.734709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.741316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.748081] Lustre: Skipped 1 previous similar message [ 1141.748430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.767222] Lustre: Skipped 1 previous similar message [ 1146.734298] Lustre: server umount lustre-OST0000 complete [ 1151.971810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.973188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.978898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.998059] Lustre: Skipped 1 previous similar message [ 1156.857366] Lustre: server umount lustre-MDT0000 complete [ 1157.090617] LustreError: 34995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.108489] LustreError: 34995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1159.811587] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936761 with bad export cookie 11918047191216784356 [ 1159.823561] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.095812] Lustre: server umount lustre-MDT0001 complete [ 1169.605693] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 1172.617775] Key type lgssc unregistered [ 1172.924603] LNet: 38552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.936650] LNetError: 38552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.959346] LNet: Removed LNI 192.168.204.149@tcp [ 1174.208727] Key type .llcrypt unregistered [ 1174.212000] Key type ._llcrypt unregistered [ 1197.981578] Key type ._llcrypt registered [ 1197.983915] Key type .llcrypt registered [ 1198.065403] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 1198.886582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.928298] alg: No test for adler32 (adler32-zlib) [ 1199.884445] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 1200.065488] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 1201.800277] Key type lgssc registered [ 1202.784759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1211.215465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1211.272407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.089211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.216761] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1223.653408] LustreError: 39966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.680521] LustreError: 39966:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1228.757409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.772269] LustreError: 39965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.170198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.167880] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1245.409923] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1252.435210] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1260.333558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1260.671141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.480399] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1274.950587] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1283.253167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1293.796376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.806681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.835643] Lustre: Skipped 1 previous similar message [ 1293.864614] Lustre: Skipped 2 previous similar messages [ 1297.042213] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 1298.414699] Lustre: server umount lustre-MDT0000 complete [ 1298.937759] LustreError: 39960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.988370] LustreError: 39960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1302.129719] LustreError: 41502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.362131] LustreError: 39947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936905 with bad export cookie 294743226887401295 [ 1303.371899] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.379516] LustreError: 39947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1304.037020] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.059522] Lustre: Skipped 1 previous similar message [ 1304.065555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1307.243988] LustreError: 39961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.256077] LustreError: 39961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1307.262810] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 1309.912142] Lustre: server umount lustre-MDT0001 complete [ 1338.849380] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1339.083733] Lustre: server umount lustre-OST0000 complete [ 1358.413395] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 1361.986584] Key type lgssc unregistered [ 1362.340830] LNet: 43852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.345646] LNetError: 43852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.377197] LNet: Removed LNI 192.168.204.149@tcp [ 1363.614094] Key type .llcrypt unregistered [ 1363.618230] Key type ._llcrypt unregistered [ 1389.028317] Key type ._llcrypt registered [ 1389.031235] Key type .llcrypt registered [ 1389.187419] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 1390.482565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1390.514209] alg: No test for adler32 (adler32-zlib) [ 1391.517289] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 1391.716841] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 1393.368605] Key type lgssc registered [ 1394.863206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1402.183637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1402.199825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.796532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.274940] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1413.092543] LustreError: 45264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.122210] LustreError: 45264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1418.215973] LustreError: 45265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.470645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.155428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1427.040130] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1435.509862] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1440.013279] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1447.197283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1448.010135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.049720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1454.979841] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1462.819634] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1475.042198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1475.050371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.060939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1475.764501] Lustre: lustre-OST0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 1477.606652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1477.614109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1477.616641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.635087] Lustre: Skipped 1 previous similar message [ 1479.458841] Lustre: server umount lustre-OST0000 complete [ 1498.029608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.056819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.580039] Lustre: server umount lustre-MDT0000 complete [ 1507.302662] LustreError: 45260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.341403] LustreError: 45260:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1508.806983] LustreError: 45246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937110 with bad export cookie 12083122772396535594 [ 1508.813075] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1508.822491] LustreError: 45246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1523.168949] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1523.626141] Lustre: server umount lustre-MDT0001 complete [ 1535.369251] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 1538.066409] Key type lgssc unregistered [ 1538.358068] LNet: 48976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.366319] LNetError: 48976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.395429] LNet: Removed LNI 192.168.204.149@tcp [ 1539.469616] Key type .llcrypt unregistered [ 1539.471515] Key type ._llcrypt unregistered [ 1567.251619] Key type ._llcrypt registered [ 1567.255711] Key type .llcrypt registered [ 1567.457689] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 1569.560957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.835636] alg: No test for adler32 (adler32-zlib) [ 1571.171748] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 1571.648441] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 1573.402507] Key type lgssc registered [ 1575.445815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1585.446605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1585.462560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.050269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1592.322787] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1597.414436] LustreError: 50390:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.456327] LustreError: 50390:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1602.542167] LustreError: 50389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.051215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.625382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1608.678787] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1616.941639] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1622.725611] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1629.834788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.359059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.432441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1637.277074] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1644.063201] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1653.784958] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1658.127684] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1670.250940] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-09 3h01m11s [ 1671.885391] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1674.728860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.748473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.828777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.844023] Lustre: Skipped 5 previous similar messages [ 1677.444949] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 1681.063662] Lustre: server umount lustre-MDT0000 complete [ 1681.896898] LustreError: 50389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.908891] LustreError: 50389:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1684.592993] LustreError: 50386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.224895] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1690.779722] LustreError: 52834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1690.802725] LustreError: 52834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1694.983053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.341544] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1695.932865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.052722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1696.955401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1701.214359] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1704.907370] Lustre: DEBUG MARKER: Wait for df (36732) ... [ 1706.682608] Lustre: DEBUG MARKER: done [ 1710.562689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1710.571896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.592616] Lustre: Skipped 2 previous similar messages [ 1710.603305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1710.610155] Lustre: Skipped 1 previous similar message [ 1711.073386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1717.737684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1717.747959] Lustre: Skipped 3 previous similar messages [ 1724.384109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1724.614643] Lustre: server umount lustre-OST0000 complete [ 1733.091523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.091876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1733.123217] Lustre: Skipped 1 previous similar message [ 1733.124654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1733.124660] Lustre: Skipped 3 previous similar messages [ 1735.949277] Lustre: server umount lustre-MDT0000 complete [ 1740.665930] LustreError: 53021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937342 with bad export cookie 8013601391686029834 [ 1740.678175] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.684738] LustreError: 53021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1741.146705] Lustre: server umount lustre-MDT0001 complete [ 1755.511687] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 1759.341586] Key type lgssc unregistered [ 1759.719594] LNet: 55331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.731053] LNetError: 55331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.754876] LNet: Removed LNI 192.168.204.149@tcp [ 1761.063093] Key type .llcrypt unregistered [ 1761.066918] Key type ._llcrypt unregistered [ 1806.825343] Key type ._llcrypt registered [ 1806.829467] Key type .llcrypt registered [ 1806.908539] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 1808.017564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1808.041642] alg: No test for adler32 (adler32-zlib) [ 1809.191586] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 1809.486108] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 1811.163057] Key type lgssc registered [ 1812.488460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1819.599491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1819.624807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.006682] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1821.031336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1821.236845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.555215] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1835.860162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.968860] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1836.001996] Lustre: 57538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1836.168120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1840.333258] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1848.175425] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1855.827522] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1863.636407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1863.828490] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1864.031624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.397779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1870.463981] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1878.782449] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1884.812104] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-09 3h04m46s [ 1886.516559] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1892.886912] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1895.545870] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1895.828695] LustreError: 57544:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1916.060757] Lustre: lustre-MDT0000: Client 22aad147-e554-48eb-90fa-ac5dcff7846b (at 192.168.204.49@tcp) reconnecting [ 1935.897322] LustreError: 57544:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1937.795331] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1940.639878] Lustre: DEBUG MARKER: done [ 1949.157242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.164341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.172099] Lustre: Skipped 1 previous similar message [ 1949.186408] Lustre: Skipped 1 previous similar message [ 1953.357926] Lustre: server umount lustre-OST0000 complete [ 1959.405092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1959.408930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.423681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.448383] Lustre: Skipped 1 previous similar message [ 1959.497120] Lustre: Skipped 1 previous similar message [ 1960.550531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.559199] Lustre: Skipped 1 previous similar message [ 1963.591146] Lustre: server umount lustre-MDT0000 complete [ 1967.487266] LustreError: 58280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937569 with bad export cookie 6753964789188963500 [ 1967.494124] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.502818] LustreError: 58280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1967.986798] Lustre: server umount lustre-MDT0001 complete [ 1980.427364] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 1984.626356] Key type lgssc unregistered [ 1984.939704] LNet: 61867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.946771] LNetError: 61867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.962904] LNet: Removed LNI 192.168.204.149@tcp [ 1985.879442] Key type .llcrypt unregistered [ 1985.881896] Key type ._llcrypt unregistered [ 2033.781374] Key type ._llcrypt registered [ 2033.786745] Key type .llcrypt registered [ 2033.930049] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2034.702023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2034.814491] alg: No test for adler32 (adler32-zlib) [ 2035.931853] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 2036.146907] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 2037.872544] Key type lgssc registered [ 2038.920734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2043.644432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2043.671413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.939996] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2044.956353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2045.067833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.764374] Lustre: Failing over lustre-MDT0000 [ 2045.941093] Lustre: server umount lustre-MDT0000 complete [ 2053.418472] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:07:34 (1788937654) [ 2059.625647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.066435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.604451] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2073.464814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.631811] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2073.687355] Lustre: 64662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2074.121792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.960895] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2087.197764] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2093.590310] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2100.370472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2100.562644] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2100.771222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2106.645941] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2110.024404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2114.976774] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2121.968603] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2130.406430] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.408476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.421821] Lustre: Skipped 2 previous similar messages [ 2130.435794] Lustre: Skipped 2 previous similar messages [ 2135.529353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.382368] Lustre: server umount lustre-MDT0000 complete [ 2140.777974] LustreError: 66522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937742 with bad export cookie 4846494955917282676 [ 2140.792605] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.138382] Lustre: server umount lustre-MDT0001 complete [ 2145.373898] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2150.940664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2160.527409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.120768] Lustre: 63583:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937747/real 1788937747] req@00000000b96fa53d x1875837072746880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788937763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2161.165785] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.282404] LustreError: 63582:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e00fe524 x1875837072747648/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2165.714295] LustreError: 66516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.727777] LustreError: 66516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2165.813354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.975716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2166.822153] Lustre: 63585:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937752/real 1788937752] req@00000000f4244d44 x1875837072747008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788937768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2166.825128] LustreError: 66517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.913989] LustreError: 66516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.933257] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2171.411782] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2172.001717] Lustre: 63585:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937758/real 1788937758] req@0000000027ce869d x1875837072747264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788937774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.059393] Lustre: 63584:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937763/real 1788937763] req@00000000e8088582 x1875837072747392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788937779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.509568] LustreError: 66517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.536877] LustreError: 66517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2181.546183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.828888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.204901] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2187.251902] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2195.473455] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2201.100515] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2212.835960] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.839726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2212.867108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2212.867116] Lustre: Skipped 2 previous similar messages [ 2214.403788] Lustre: server umount lustre-MDT0000 complete [ 2217.957128] LustreError: 66516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.983287] LustreError: 66516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2218.470448] LustreError: 64654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937820 with bad export cookie 4846494955917284958 [ 2218.485027] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.491336] LustreError: 64654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2219.043607] Lustre: server umount lustre-MDT0001 complete [ 2225.817244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2236.343613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.459346] Lustre: 63584:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937825/real 1788937825] req@000000002d7270b6 x1875837072774912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788937841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2239.510092] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.526701] Lustre: Skipped 2 previous similar messages [ 2244.011051] LustreError: 66516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.106890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.165352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2249.120470] Lustre: 63585:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937835/real 1788937835] req@0000000097b8a7b6 x1875837072775296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788937851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2249.190103] Lustre: 63585:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2250.014499] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2260.758705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.615774] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2267.254024] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2268.720956] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2277.133432] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2283.880562] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2291.444638] Lustre: server umount lustre-MDT0000 complete [ 2292.193659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.208062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.256367] LustreError: 66516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.295095] LustreError: 66516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2295.346824] LustreError: 64652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937897 with bad export cookie 4846494955917286337 [ 2295.349114] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.357413] LustreError: 64652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2295.643637] Lustre: server umount lustre-MDT0001 complete [ 2301.986696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2305.261839] Lustre: DEBUG MARKER: files compared the same [ 2317.709656] Lustre: server umount lustre-OST0000 complete [ 2334.264701] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 2337.571458] Key type lgssc unregistered [ 2337.894188] LNet: 74399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.912158] LNetError: 74399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.961346] LNet: Removed LNI 192.168.204.149@tcp [ 2338.896458] Key type .llcrypt unregistered [ 2338.901556] Key type ._llcrypt unregistered [ 2364.019990] Key type ._llcrypt registered [ 2364.023909] Key type .llcrypt registered [ 2364.183774] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2365.669876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2365.706941] alg: No test for adler32 (adler32-zlib) [ 2366.889624] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 2367.198781] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 2368.906241] Key type lgssc registered [ 2370.279652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2378.162189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2378.218178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.816315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.931089] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug +malloc all [ 2388.961722] LustreError: 75810:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.985903] LustreError: 75810:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2393.998773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.086115] LustreError: 75811:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.370136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2400.027444] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug +malloc all [ 2408.937794] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2415.342747] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2422.767775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.167571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.249769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2428.203915] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug +malloc all [ 2435.930949] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2449.889866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.891104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.898487] Lustre: Skipped 1 previous similar message [ 2449.915898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2453.859528] Lustre: server umount lustre-OST0000 complete [ 2460.134421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.134986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2460.155465] Lustre: Skipped 1 previous similar message [ 2460.172916] Lustre: Skipped 2 previous similar messages [ 2463.890403] Lustre: server umount lustre-MDT0000 complete [ 2465.254622] LustreError: 75806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.287174] LustreError: 75806:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2467.303700] LustreError: 78589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938069 with bad export cookie 14317318450654099919 [ 2467.308492] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2467.322558] LustreError: 78589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2467.651717] Lustre: server umount lustre-MDT0001 complete [ 2478.922204] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 2481.656338] Key type lgssc unregistered [ 2481.987867] LNet: 79340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2482.000628] LNetError: 79340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2483.046788] LNet: Removed LNI 192.168.204.149@tcp [ 2483.843487] Key type .llcrypt unregistered [ 2483.846507] Key type ._llcrypt unregistered [ 2508.639743] Key type ._llcrypt registered [ 2508.642197] Key type .llcrypt registered [ 2508.775436] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2510.069912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2510.091266] alg: No test for adler32 (adler32-zlib) [ 2511.117230] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 2511.479792] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 2513.226109] Key type lgssc registered [ 2514.240658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2522.786683] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2522.803163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2538.400689] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2543.520686] LustreError: 80728:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.149@tcp: failed processing log, type 4: rc = -110 [ 2570.272375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.797512] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2584.304562] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2592.018540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.168787] Lustre: *** cfs_fail_loc=706, val=0*** [ 2592.596513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2592.733153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2597.370953] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2597.871782] LustreError: 80753:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.897723] LustreError: 80753:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2602.986568] LustreError: 80754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.042429] LustreError: 81813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.062080] LustreError: 81813:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2606.201509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.527436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2610.844614] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2618.762300] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2625.146441] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2628.577625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2628.586299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.599164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.163967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.168038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2632.182976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.727725] Lustre: server umount lustre-OST0000 complete [ 2637.283535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2637.283796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.304071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.317244] Lustre: Skipped 1 previous similar message [ 2637.329647] Lustre: Skipped 2 previous similar messages [ 2642.402496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.413794] Lustre: Skipped 1 previous similar message [ 2643.489557] Lustre: server umount lustre-MDT0000 complete [ 2647.136561] LustreError: 80736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938249 with bad export cookie 4345490929660577650 [ 2647.149186] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.152121] LustreError: 80736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2647.559776] Lustre: server umount lustre-MDT0001 complete [ 2660.709921] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 2663.792640] Key type lgssc unregistered [ 2664.103165] LNet: 84342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2664.113040] LNetError: 84342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2665.143347] LNet: Removed LNI 192.168.204.149@tcp [ 2666.227374] Key type .llcrypt unregistered [ 2666.230462] Key type ._llcrypt unregistered [ 2695.511823] Key type ._llcrypt registered [ 2695.513826] Key type .llcrypt registered [ 2695.608813] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2696.772368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2696.998099] alg: No test for adler32 (adler32-zlib) [ 2698.071811] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 2698.444876] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 2700.160548] Key type lgssc registered [ 2701.301884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2709.305904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2709.322458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.578651] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2723.005303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.548599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2728.200936] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2728.932355] LustreError: 86443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.945055] LustreError: 86443:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2733.031191] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.064764] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2736.178672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.383647] LustreError: 86443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2736.396774] LustreError: 86443:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2736.421255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.730718] LustreError: 86446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.750678] LustreError: 86446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2742.594549] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2746.852810] LustreError: 86441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.864602] LustreError: 86441:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2751.322520] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.337574] Lustre: MGS: Client 18115016-5425-4840-8243-3b1515bb5d1d (at 0@lo) reconnecting [ 2751.361429] LustreError: 86457:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2751.599201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2751.669924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2755.860468] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2771.938098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2771.945665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.955435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2772.454517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2772.458845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.465351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2772.468621] Lustre: Skipped 1 previous similar message [ 2776.893890] Lustre: server umount lustre-OST0000 complete [ 2782.689391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.691149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2782.712526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.211983] Lustre: server umount lustre-MDT0000 complete [ 2787.811338] LustreError: 87986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.823286] LustreError: 87986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2790.622347] Lustre: server umount lustre-MDT0001 complete [ 2794.910311] Lustre: server umount MGS complete [ 2809.005058] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:20:09 (1788938409) [ 2840.440720] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 2850.188519] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2887.465270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.160356] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2902.464127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.787352] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2902.839612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2902.956166] Lustre: lustre-MDT0001: new disk, initializing [ 2903.044462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2903.094225] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2903.100907] Lustre: 93636:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2903.114534] LustreError: 93636:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2907.603527] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2917.411021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2917.675399] Lustre: lustre-OST0000: new disk, initializing [ 2917.688826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2917.703208] Lustre: 94403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2917.807960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.253633] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2926.113400] Lustre: 94795:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000dfac8b5a x1875837766901632/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788938544 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2933.764875] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2940.688873] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2940.701836] Lustre: MGS: Client 9f30c73d-a3e6-4a66-8e1d-6ba4fbbde823 (at 0@lo) reconnecting [ 2940.739743] Lustre: 92901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2940.752317] LustreError: 93621:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2940.760297] LustreError: 93621:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2940.781379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2940.840391] Lustre: lustre-MDT0000: new disk, initializing [ 2940.936598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.959390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2940.989160] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2942.068195] Lustre: 95471:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000064549c1a x1875837766908160/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788938560 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2945.785588] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2947.233657] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2951.149315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2951.156243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2951.301777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2977.252158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2977.259311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.277980] Lustre: Skipped 1 previous similar message [ 2977.286558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2977.299419] Lustre: Skipped 1 previous similar message [ 2983.393523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2983.414577] Lustre: Skipped 3 previous similar messages [ 2990.048231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2990.258726] Lustre: server umount lustre-OST0000 complete [ 2993.634810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2993.636333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2993.659406] Lustre: Skipped 1 previous similar message [ 2993.668149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2993.676153] Lustre: Skipped 3 previous similar messages [ 2999.912530] Lustre: server umount lustre-MDT0000 complete [ 3009.369202] Lustre: server umount lustre-MDT0001 complete [ 3013.856774] Lustre: server umount MGS complete [ 3026.565273] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:23:47 (1788938627) [ 3044.829150] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 3049.013100] Key type lgssc unregistered [ 3049.404992] LNet: 97967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3049.414134] LNetError: 97967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3049.448066] LNet: Removed LNI 192.168.204.149@tcp [ 3050.821132] Key type .llcrypt unregistered [ 3050.822437] Key type ._llcrypt unregistered [ 3072.467761] Key type ._llcrypt registered [ 3072.472461] Key type .llcrypt registered [ 3072.655556] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 3073.771270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.862129] alg: No test for adler32 (adler32-zlib) [ 3075.176950] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 3075.443960] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 3077.088619] Key type lgssc registered [ 3078.366165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3085.601866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3085.617921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.060502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.176077] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3097.576743] LustreError: 99186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3097.596878] LustreError: 99186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3101.289755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.782697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3106.577273] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3114.997634] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3121.026318] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3127.954669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3128.291966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.413698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3134.734232] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3142.104874] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3156.973180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.974125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3156.985116] Lustre: Skipped 1 previous similar message [ 3156.986083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3159.627592] Lustre: server umount lustre-OST0000 complete [ 3167.203038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.205786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3167.214786] Lustre: Skipped 1 previous similar message [ 3167.226339] Lustre: Skipped 1 previous similar message [ 3170.263301] Lustre: server umount lustre-MDT0000 complete [ 3172.324890] LustreError: 99187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.360327] LustreError: 99187:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3174.259142] LustreError: 99168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938776 with bad export cookie 2790474003642974708 [ 3174.264257] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3174.290728] LustreError: 99168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3174.809574] Lustre: server umount lustre-MDT0001 complete [ 3187.587737] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 3191.238666] Key type lgssc unregistered [ 3191.670945] LNet: 102723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3191.680028] LNetError: 102723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3191.706255] LNet: Removed LNI 192.168.204.149@tcp [ 3193.057718] Key type .llcrypt unregistered [ 3193.062826] Key type ._llcrypt unregistered [ 3211.065486] Key type ._llcrypt registered [ 3211.066993] Key type .llcrypt registered [ 3211.210224] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 3212.755489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3212.813162] alg: No test for adler32 (adler32-zlib) [ 3213.865568] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 3214.087808] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 3215.800219] Key type lgssc registered [ 3217.299929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3224.489624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3224.496913] LustreError: 103800:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3224.519649] LustreError: 103803:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3224.524786] LustreError: 103800:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 3224.533780] LustreError: 103800:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3224.539866] LustreError: 103800:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3224.551498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.222646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.501500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.889492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3241.496323] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3248.157411] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing lsmod [ 3251.606572] LustreError: 105006:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3251.637943] LustreError: 105008:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3251.648492] LustreError: 105006:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 3251.656982] LustreError: 105006:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3251.678224] LustreError: 105006:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3251.686486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3252.113576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.293430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3256.308908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.343424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3257.340265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.342936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3257.370896] Lustre: Skipped 1 previous similar message [ 3257.394313] Lustre: Skipped 2 previous similar messages [ 3260.463355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3260.468650] Lustre: Skipped 2 previous similar messages [ 3265.514985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3265.529619] Lustre: Skipped 2 previous similar messages [ 3270.112145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3270.626529] LustreError: 103840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.662495] LustreError: 103840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3272.379298] Lustre: server umount lustre-MDT0000 complete [ 3275.748350] LustreError: 105043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.775983] LustreError: 105043:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3277.080437] LustreError: 103816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938879 with bad export cookie 16558942735920371085 [ 3277.096608] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3278.944692] LustreError: 104271:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f7093af8 x1875838307950592/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3279.204144] Lustre: server umount lustre-MDT0001 complete [ 3294.671342] Lustre: server umount lustre-OST0000 complete [ 3305.407097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.835894] LustreError: 106078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.922087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.075900] LustreError: 106079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.219793] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3315.197340] LustreError: 106078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3322.659662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.260213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3329.712801] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3339.305790] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3346.121518] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3353.927038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3354.581861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.941771] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3370.557742] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3375.108244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3384.297291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3384.305403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.323448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3385.325500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3387.622390] Lustre: server umount lustre-OST0000 complete [ 3395.051726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3395.069591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.086692] Lustre: Skipped 1 previous similar message [ 3395.096402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3395.107386] Lustre: Skipped 2 previous similar messages [ 3406.310697] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3406.585556] Lustre: server umount lustre-MDT0000 complete [ 3410.935371] LustreError: 106060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939013 with bad export cookie 16558942735920371785 [ 3410.936608] LustreError: 107903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.943373] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3410.958208] LustreError: 106060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3411.019107] LustreError: 107903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3411.390861] Lustre: server umount lustre-MDT0001 complete [ 3423.748967] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 3426.977828] Key type lgssc unregistered [ 3427.371058] LNet: 109613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3427.383737] LNetError: 109613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3427.401945] LNet: Removed LNI 192.168.204.149@tcp [ 3428.336808] Key type .llcrypt unregistered [ 3428.339994] Key type ._llcrypt unregistered [ 3454.945666] Key type ._llcrypt registered [ 3454.952806] Key type .llcrypt registered [ 3455.106878] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 3456.538462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3456.605931] alg: No test for adler32 (adler32-zlib) [ 3457.755572] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 3457.968905] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 3459.632238] Key type lgssc registered [ 3461.212250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3469.646879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3469.662760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.287370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.859727] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3481.573688] LustreError: 111029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.617577] LustreError: 111029:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3486.702026] LustreError: 111030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3487.285112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.792772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3494.461646] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3505.080149] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3511.870475] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3519.974670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3520.477329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3527.445822] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3537.200141] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3542.039863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3548.049919] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3551.727538] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3557.351560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3557.355422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.358825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3557.384372] Lustre: Skipped 1 previous similar message [ 3561.782339] Lustre: server umount lustre-OST0000 complete [ 3567.597312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.608561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3567.618883] Lustre: Skipped 1 previous similar message [ 3567.629141] Lustre: Skipped 1 previous similar message [ 3571.525442] Lustre: server umount lustre-MDT0000 complete [ 3572.713436] LustreError: 111030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3572.734221] LustreError: 111030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3575.754733] LustreError: 111011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939177 with bad export cookie 17103557147493642217 [ 3575.764787] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3575.767064] LustreError: 111011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3576.104345] Lustre: server umount lustre-MDT0001 complete [ 3586.867872] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 3589.594930] Key type lgssc unregistered [ 3589.880663] LNet: 114667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3589.893320] LNetError: 114667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3589.919399] LNet: Removed LNI 192.168.204.149@tcp [ 3591.415518] Key type .llcrypt unregistered [ 3591.418045] Key type ._llcrypt unregistered [ 3616.330457] Key type ._llcrypt registered [ 3616.333294] Key type .llcrypt registered [ 3616.554354] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 3617.846843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3617.894791] alg: No test for adler32 (adler32-zlib) [ 3618.971979] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 3619.214750] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 3620.865121] Key type lgssc registered [ 3622.096920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3632.088111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3632.108386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.626764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3639.545319] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3643.906154] LustreError: 115886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3643.943475] LustreError: 115886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3648.997777] LustreError: 115887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3649.799253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3650.332390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3655.996128] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3665.703293] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3673.136861] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3680.507317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3680.870703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.556586] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3696.124584] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3703.436637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3719.144180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.171716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3719.172627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3719.196366] Lustre: Skipped 1 previous similar message [ 3720.211875] Lustre: server umount lustre-OST0000 complete [ 3727.340894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3727.355328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3727.386080] Lustre: Skipped 1 previous similar message [ 3727.397517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3729.385722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3729.392253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3732.095917] Lustre: server umount lustre-MDT0000 complete [ 3734.503630] LustreError: 116837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3734.520714] LustreError: 116837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3736.872258] LustreError: 115867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939338 with bad export cookie 5547983997880430244 [ 3736.872955] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3736.886874] LustreError: 115867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3737.599390] Lustre: server umount lustre-MDT0001 complete [ 3752.757796] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 3756.273569] Key type lgssc unregistered [ 3756.624567] LNet: 119443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3756.633978] LNetError: 119443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3756.671920] LNet: Removed LNI 192.168.204.149@tcp [ 3757.847764] Key type .llcrypt unregistered [ 3757.853969] Key type ._llcrypt unregistered [ 3784.760334] Key type ._llcrypt registered [ 3784.763207] Key type .llcrypt registered [ 3784.857701] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 3786.322149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3786.382767] alg: No test for adler32 (adler32-zlib) [ 3787.371405] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 3787.519799] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 3789.168920] Key type lgssc registered [ 3790.726168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3800.230725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3800.263873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.041624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.997489] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3812.335692] LustreError: 120860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.369103] LustreError: 120860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3813.355891] LustreError: 120861:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3818.029054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3818.588106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3823.749606] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3831.222194] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3838.587800] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3847.946070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3848.168690] LustreError: 122680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.670920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.831236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3856.701098] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3867.332651] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3875.769408] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3880.600044] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3880.609836] Lustre: Skipped 2 previous similar messages [ 3885.692357] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3885.755350] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3885.762824] Lustre: Skipped 1 previous similar message [ 3886.258249] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3886.267387] Lustre: Skipped 2 previous similar messages [ 3895.841934] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3895.844227] Lustre: Skipped 1 previous similar message [ 3902.993126] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.49@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3903.011575] Lustre: Skipped 2 previous similar messages [ 3904.898734] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.49@tcp 192.168.0.[2,10]@tcp [ 3915.927131] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.49/24@tcp in log lustre-MDT0000 [ 3915.947306] Lustre: Skipped 2 previous similar messages [ 3925.050591] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.49/24@tcp [ 3925.059145] Lustre: Skipped 1 previous similar message [ 3929.581209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3929.582361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.594960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3929.627641] Lustre: Skipped 1 previous similar message [ 3931.826852] Lustre: server umount lustre-OST0000 complete [ 3939.814191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.816149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3939.843721] Lustre: Skipped 1 previous similar message [ 3939.860546] Lustre: Skipped 1 previous similar message [ 3943.511701] Lustre: server umount lustre-MDT0000 complete [ 3944.934068] LustreError: 120860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.965517] LustreError: 120860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3948.199612] LustreError: 124602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939550 with bad export cookie 15838755203768763650 [ 3948.203081] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.221174] LustreError: 124602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3948.819939] Lustre: server umount lustre-MDT0001 complete [ 3963.440690] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 3967.092049] Key type lgssc unregistered [ 3967.459271] LNet: 125903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3967.474687] LNetError: 125903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3967.500180] LNet: Removed LNI 192.168.204.149@tcp [ 3968.682716] Key type .llcrypt unregistered [ 3968.685923] Key type ._llcrypt unregistered [ 4001.522263] Key type ._llcrypt registered [ 4001.528504] Key type .llcrypt registered [ 4001.697917] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 4002.862140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4002.973924] alg: No test for adler32 (adler32-zlib) [ 4004.078363] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4004.424674] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 4006.160284] Key type lgssc registered [ 4007.344462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4015.873619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4015.899728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.400258] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4017.428236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4017.450044] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.49/24@tcp [ 4022.605928] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4027.892493] LustreError: 127461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.920027] LustreError: 127461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4031.843413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4032.293638] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4032.338540] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.49/24@tcp [ 4032.411341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4037.941955] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4046.163218] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4052.513941] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4059.654271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4060.025144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.072383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4066.720463] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4074.326078] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4087.781682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.782376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4087.788080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4087.790884] LustreError: Skipped 2 previous similar messages [ 4087.798783] Lustre: Skipped 1 previous similar message [ 4092.904210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4092.909070] Lustre: Skipped 1 previous similar message [ 4093.609331] Lustre: server umount lustre-OST0000 complete [ 4098.018612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.032440] Lustre: Skipped 1 previous similar message [ 4098.039124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4103.138025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4103.150068] Lustre: Skipped 2 previous similar messages [ 4108.270569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4108.285828] Lustre: Skipped 1 previous similar message [ 4112.352185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4112.813595] Lustre: server umount lustre-MDT0000 complete [ 4113.381517] LustreError: 127460:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.392339] LustreError: 127460:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4117.305371] LustreError: 127441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939719 with bad export cookie 15890199155333514687 [ 4117.317081] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4117.891155] Lustre: server umount lustre-MDT0001 complete [ 4130.583990] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 4133.615889] Key type lgssc unregistered [ 4133.930263] LNet: 131052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4133.939754] LNetError: 131052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4133.970683] LNet: Removed LNI 192.168.204.149@tcp [ 4135.051387] Key type .llcrypt unregistered [ 4135.055186] Key type ._llcrypt unregistered [ 4170.388742] Key type ._llcrypt registered [ 4170.390547] Key type .llcrypt registered [ 4170.472948] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 4187.583992] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 4189.359925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4189.573059] alg: No test for adler32 (adler32-zlib) [ 4190.767432] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4191.044809] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 4192.888292] Key type lgssc registered [ 4194.379457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4240.374700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4240.421941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.796739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4241.850685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4241.933802] Lustre: lustre-MDT0000: new disk, initializing [ 4242.014701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.028110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4247.542838] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4260.062384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.235196] Lustre: 134450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4260.336782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4260.341058] Lustre: Skipped 1 previous similar message [ 4260.497695] Lustre: lustre-MDT0001: new disk, initializing [ 4260.659452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4260.737238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4260.752762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4266.604895] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4278.803487] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4284.304866] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4291.929301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4292.107818] Lustre: lustre-OST0000: new disk, initializing [ 4292.111349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4292.116357] Lustre: 136418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4292.190333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4298.633454] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4301.867359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4301.885230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4302.004735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4308.860933] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4313.502918] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4313.706519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4317.933509] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4318.155494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4322.274425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4322.283165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.293154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4322.792588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.812414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4322.823988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4326.641785] Lustre: server umount lustre-OST0000 complete [ 4332.513337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4332.525684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.549551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.145529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.151897] Lustre: Skipped 3 previous similar messages [ 4343.265823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4343.275835] Lustre: Skipped 1 previous similar message [ 4345.312242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4345.435176] Lustre: server umount lustre-MDT0000 complete [ 4348.794393] LustreError: 134442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939950 with bad export cookie 13913870974928247594 [ 4348.804114] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4348.809181] LustreError: 134442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4349.073855] Lustre: server umount lustre-MDT0001 complete [ 4356.283281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.572590] LustreError: 138636:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.589432] LustreError: 138636:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4356.637787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4360.949961] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4361.698065] LustreError: 138637:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.821424] LustreError: 138636:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.077347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.318427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4373.688838] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4383.138770] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4392.487205] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4400.203710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4400.583650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4405.945658] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4413.900720] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4419.371595] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4419.623688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4424.501717] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4424.687361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4433.287212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4433.478152] Lustre: lustre-OST0001: new disk, initializing [ 4433.483040] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4433.490146] Lustre: 142012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4433.587658] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4439.885477] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4443.767446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4443.785268] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4443.924144] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4448.219706] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4448.408154] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4452.569743] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4452.860381] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4458.398517] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95e882fc8000.ost_server_uuid 50 [ 4459.793573] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95e882fc8000.ost_server_uuid in FULL state after 0 sec [ 4469.741869] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4469.752889] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.772595] Lustre: Skipped 1 previous similar message [ 4469.782823] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4469.805200] Lustre: Skipped 1 previous similar message [ 4473.458984] Lustre: server umount lustre-OST0001 complete [ 4474.868407] LustreError: 140461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4479.972112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.980067] LustreError: 140461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4479.986015] Lustre: Skipped 2 previous similar messages [ 4480.020777] LustreError: 140461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4484.900702] Lustre: server umount lustre-OST0000 complete [ 4490.210118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.214313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4490.242190] Lustre: Skipped 1 previous similar message [ 4490.269677] Lustre: Skipped 3 previous similar messages [ 4496.364918] Lustre: server umount lustre-MDT0000 complete [ 4500.449576] LustreError: 139366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.483269] LustreError: 139366:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4502.248933] LustreError: 138616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940104 with bad export cookie 13913870974928248476 [ 4502.251566] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4502.259959] LustreError: 138616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4502.559234] Lustre: server umount lustre-MDT0001 complete [ 4523.660602] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 4526.504543] Key type lgssc unregistered [ 4526.866941] LNet: 145016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4526.873197] LNetError: 145016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4526.889985] LNet: Removed LNI 192.168.204.149@tcp [ 4527.740618] Key type .llcrypt unregistered [ 4527.744628] Key type ._llcrypt unregistered [ 4574.217927] Key type ._llcrypt registered [ 4574.221028] Key type .llcrypt registered [ 4574.325748] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 4588.510450] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 4590.185727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4590.301286] alg: No test for adler32 (adler32-zlib) [ 4591.755168] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4592.244741] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 4594.080316] Key type lgssc registered [ 4595.319257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4635.209509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4635.237552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.718559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4636.820756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4636.897149] Lustre: lustre-MDT0000: new disk, initializing [ 4637.037703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4637.089942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4644.571063] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4657.008583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.105299] Lustre: 149166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4657.166593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4657.170927] Lustre: Skipped 1 previous similar message [ 4657.225968] Lustre: lustre-MDT0001: new disk, initializing [ 4657.379540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4657.441900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4657.464436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4663.236021] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4674.014120] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4678.592773] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4685.582350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.035626] Lustre: lustre-OST0000: new disk, initializing [ 4686.039451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4686.044878] Lustre: 151142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4686.185568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.733891] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4696.207167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4696.281899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4696.519936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4705.623321] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4711.209450] Lustre: Failing over lustre-OST0000 [ 4711.349644] Lustre: server umount lustre-OST0000 complete [ 4727.249184] Lustre: 147390:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788940313/real 1788940313] req@00000000dfdda9dd x1875839752897408/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788940329 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4727.282524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.313261] Lustre: 147389:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788940315/real 1788940315] req@000000006ae41457 x1875839752897920/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788940331 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4729.324130] Lustre: 147389:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4729.519738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4729.749215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4729.766185] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4733.792202] Lustre: 147390:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788940319/real 1788940319] req@0000000009224b14 x1875839752898944/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788940335 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4733.840289] Lustre: 147390:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4735.224765] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4737.908876] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4738.913380] Lustre: 147392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788940324/real 1788940324] req@00000000bab657ef x1875839752900224/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788940340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4738.955212] Lustre: 147392:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4739.621028] Lustre: Failing over lustre-MDT0000 [ 4739.647396] LustreError: 147388:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4739.734332] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 4739.743257] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4739.747109] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4740.176760] Lustre: server umount lustre-MDT0000 complete [ 4743.008197] Lustre: 147391:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788940329/real 1788940329] req@000000006ccdb50e x1875839752901248/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788940345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4743.139671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.163560] Lustre: Skipped 2 previous similar messages [ 4748.257678] LustreError: 150321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.287535] LustreError: 150321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4749.936318] LustreError: 149173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.380943] LustreError: 149172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.407047] LustreError: 149172:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4758.503523] LustreError: 149177:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4758.552571] LustreError: 149177:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4759.520422] Lustre: 147391:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788940345/real 1788940345] req@00000000322052c5 x1875839752909696/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788940361 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4759.540163] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4760.453377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.647784] LustreError: 149560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4764.677315] LustreError: 149560:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4770.103130] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4770.258386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4770.320956] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4770.441901] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4775.015787] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4775.416335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4775.458126] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4775.518275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4785.638371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.651232] Lustre: Skipped 2 previous similar messages [ 4785.651303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4785.667471] Lustre: Skipped 1 previous similar message [ 4789.733128] Lustre: server umount lustre-OST0000 complete [ 4795.882901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.885481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4795.901505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4799.712631] Lustre: server umount lustre-MDT0000 complete [ 4800.999131] LustreError: 150321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4801.017412] LustreError: 150321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4804.308480] LustreError: 149159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940406 with bad export cookie 8138667294075017498 [ 4804.316130] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4804.323622] LustreError: 149159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4804.845161] Lustre: server umount lustre-MDT0001 complete [ 4820.453877] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 4824.383182] Key type lgssc unregistered [ 4824.916322] LNet: 154794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4824.929326] LNetError: 154794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4825.963206] LNet: Removed LNI 192.168.204.149@tcp [ 4826.843273] Key type .llcrypt unregistered [ 4826.845109] Key type ._llcrypt unregistered [ 4866.836659] Key type ._llcrypt registered [ 4866.843644] Key type .llcrypt registered [ 4866.964328] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 4868.046770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4868.279968] alg: No test for adler32 (adler32-zlib) [ 4869.484441] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4869.733802] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 4871.408205] Key type lgssc registered [ 4872.948891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4882.157852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4882.190211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.972194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4889.745184] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4893.176543] LustreError: 156312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4893.199802] LustreError: 156312:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4898.281826] LustreError: 156313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4900.468000] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4903.395589] LustreError: 156312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.696808] LustreError: 156341:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4904.708422] LustreError: 156341:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4904.714450] LustreError: 156341:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4904.981092] Lustre: server umount lustre-MDT0000 complete [ 4921.486273] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 4932.498411] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 4972.776228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4973.045536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4973.088810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4973.185097] Lustre: lustre-MDT0000: new disk, initializing [ 4973.288711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.318614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4978.793876] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 4990.326914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.509275] Lustre: 159647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4990.619175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4990.626063] Lustre: Skipped 1 previous similar message [ 4990.767611] Lustre: lustre-MDT0001: new disk, initializing [ 4990.958498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4991.035472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4991.064248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4996.933779] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5006.457944] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5012.995960] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5022.993770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5023.370830] Lustre: lustre-OST0000: new disk, initializing [ 5023.374405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5023.389930] Lustre: 161615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5023.543772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5030.528291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5030.532715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5030.680499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5031.082594] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5043.442143] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5048.671626] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5050.555444] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:57:31 (1788940651) [ 5317.600882] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.602047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5317.617582] Lustre: Skipped 2 previous similar messages [ 5317.635814] Lustre: Skipped 2 previous similar messages [ 5321.786684] Lustre: server umount lustre-MDT0000 complete [ 5322.721726] LustreError: 159657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5322.743103] LustreError: 159657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5326.253107] LustreError: 159637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940928 with bad export cookie 10371336570109489736 [ 5326.259865] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5326.264900] LustreError: 159637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5327.841570] LustreError: 159651:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.844758] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5327.882940] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5331.873193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5332.822546] Lustre: server umount lustre-MDT0001 complete [ 5351.393869] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5351.771656] Lustre: server umount lustre-OST0000 complete [ 5364.849287] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 5376.376915] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 5415.995272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.281039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5416.322510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5416.403641] Lustre: lustre-MDT0000: new disk, initializing [ 5416.524247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5416.549911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5421.263368] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5434.236654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.411447] Lustre: 166294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5434.546681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5434.555328] Lustre: Skipped 1 previous similar message [ 5434.681254] Lustre: lustre-MDT0001: new disk, initializing [ 5434.882530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5434.937495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5434.959436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5440.731277] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5452.211985] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5458.688881] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5466.956879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5467.523985] Lustre: lustre-OST0000: new disk, initializing [ 5467.528976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5467.536229] Lustre: 168261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5467.610452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5474.469795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5474.497505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5474.721107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5475.788320] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5486.271656] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5491.243895] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5491.414269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5497.329583] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5497.646390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5500.389296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.426045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5500.457425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5505.512916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5505.521074] Lustre: Skipped 2 previous similar messages [ 5514.208605] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5514.538349] Lustre: server umount lustre-OST0000 complete [ 5520.870297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.879910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5520.888293] Lustre: Skipped 2 previous similar messages [ 5520.902890] Lustre: Skipped 3 previous similar messages [ 5524.617209] Lustre: server umount lustre-MDT0000 complete [ 5525.988484] LustreError: 166299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5526.039063] LustreError: 166299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5528.944468] LustreError: 166285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941131 with bad export cookie 10371336570109648573 [ 5528.952676] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5528.960098] LustreError: 166285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5529.439801] Lustre: server umount lustre-MDT0001 complete [ 5538.344992] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:05:39 (1788941139) [ 5557.633148] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 5569.188896] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 5607.870715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5608.226742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5608.305666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5608.428518] Lustre: lustre-MDT0000: new disk, initializing [ 5608.574041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.618860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5612.991878] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5625.430143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5625.629540] Lustre: 173019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5626.054268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5631.538766] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5641.992316] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5648.420640] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5656.194812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5656.709370] Lustre: lustre-OST0000: new disk, initializing [ 5656.721989] Lustre: Skipped 1 previous similar message [ 5656.727765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5656.732536] Lustre: Skipped 2 previous similar messages [ 5656.743658] Lustre: 174997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5662.969823] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5665.880211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5665.897453] Lustre: Skipped 1 previous similar message [ 5665.910382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5666.037916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5671.384441] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5681.633484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5681.637057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5681.649583] LustreError: Skipped 1 previous similar message [ 5681.651994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5681.699169] Lustre: Skipped 1 previous similar message [ 5686.673848] Lustre: server umount lustre-OST0000 complete [ 5691.876199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.897658] Lustre: Skipped 1 previous similar message [ 5697.663100] Lustre: server umount lustre-MDT0000 complete [ 5701.105368] LustreError: 173031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.130310] LustreError: 173031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5702.154571] LustreError: 173010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941304 with bad export cookie 10371336570109649455 [ 5702.168379] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5702.178715] LustreError: 173010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5702.661709] Lustre: server umount lustre-MDT0001 complete [ 5712.633079] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:08:33 (1788941313) [ 5730.977675] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 5741.170964] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 5780.073619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.498622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5780.520760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5780.581571] Lustre: lustre-MDT0000: new disk, initializing [ 5780.653474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5780.658303] Lustre: Skipped 2 previous similar messages [ 5780.680583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5785.678296] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5796.416228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5796.535302] Lustre: 179320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5796.797226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5800.157071] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5809.910164] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5816.185091] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5823.007811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.192286] Lustre: 181300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5828.837967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5829.695797] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5839.021259] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5849.577403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5849.582256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.592394] LustreError: Skipped 1 previous similar message [ 5849.597597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5849.619611] Lustre: Skipped 1 previous similar message [ 5849.656797] Lustre: Skipped 6 previous similar messages [ 5851.872691] Lustre: server umount lustre-OST0000 complete [ 5858.276100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5862.848100] Lustre: server umount lustre-MDT0000 complete [ 5864.943346] LustreError: 179330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.986650] LustreError: 179330:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5867.800752] LustreError: 179313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941469 with bad export cookie 10371336570109650540 [ 5867.802773] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5867.817128] LustreError: 179313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5868.365212] Lustre: server umount lustre-MDT0001 complete [ 5879.251083] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 5882.227795] Key type lgssc unregistered [ 5882.595975] LNet: 183153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5882.603965] LNetError: 183153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5882.619774] LNet: Removed LNI 192.168.204.149@tcp [ 5883.578798] Key type .llcrypt unregistered [ 5883.583517] Key type ._llcrypt unregistered [ 5907.711862] Key type ._llcrypt registered [ 5907.716752] Key type .llcrypt registered [ 5907.922456] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 5908.956053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5909.053977] alg: No test for adler32 (adler32-zlib) [ 5910.215691] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 5910.440471] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 5912.112570] Key type lgssc registered [ 5913.501691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5920.104407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5920.139465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.470808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.975552] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5930.978673] LustreError: 184567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5930.999884] LustreError: 184567:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5934.672955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5935.181841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5940.286164] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5948.834888] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5955.279126] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5962.732557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5963.070529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.145510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5969.200646] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5977.668474] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6001.126559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6001.126653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.131257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6001.162746] Lustre: Skipped 1 previous similar message [ 6003.472956] Lustre: server umount lustre-OST0000 complete [ 6011.364104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6011.371154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6011.373435] Lustre: Skipped 1 previous similar message [ 6014.437785] Lustre: server umount lustre-MDT0000 complete [ 6016.489358] LustreError: 184563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6016.523901] LustreError: 184563:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6018.228501] LustreError: 184548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941620 with bad export cookie 1689760571649236021 [ 6018.230176] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6018.246769] LustreError: 184548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6018.708574] Lustre: server umount lustre-MDT0001 complete [ 6028.749707] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 6031.386301] Key type lgssc unregistered [ 6031.755536] LNet: 188122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6031.765206] LNetError: 188122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6031.780727] LNet: Removed LNI 192.168.204.149@tcp [ 6032.727927] Key type .llcrypt unregistered [ 6032.729628] Key type ._llcrypt unregistered [ 6057.865508] Key type ._llcrypt registered [ 6057.869382] Key type .llcrypt registered [ 6057.940620] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 6059.127575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6059.213203] alg: No test for adler32 (adler32-zlib) [ 6060.269216] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 6060.567944] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 6062.264256] Key type lgssc registered [ 6063.770781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6071.154483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6071.170649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6072.643077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.324268] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6082.026974] LustreError: 189535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6082.062159] LustreError: 189535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6087.141780] LustreError: 189534:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6087.526417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.264501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6094.831490] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6103.421559] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6109.710675] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6116.672304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6117.018946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6118.078980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6123.997708] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6131.303162] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6138.855193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6138.859529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6138.860036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6138.876548] Lustre: Skipped 1 previous similar message [ 6138.900027] Lustre: Skipped 1 previous similar message [ 6139.527861] Lustre: lustre-OST0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 6143.985811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6144.002890] Lustre: Skipped 1 previous similar message [ 6144.370678] Lustre: server umount lustre-OST0000 complete [ 6151.608166] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff95e886099800.ost_server_uuid 50 [ 6153.273286] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6167.342292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.345923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6167.364303] Lustre: Skipped 1 previous similar message [ 6173.706222] Lustre: server umount lustre-MDT0000 complete [ 6174.690972] LustreError: 190455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.707478] LustreError: 190455:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6177.286197] LustreError: 189515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941779 with bad export cookie 14847298678829942601 [ 6177.300855] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6177.302243] LustreError: 189515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6177.625735] Lustre: server umount lustre-MDT0001 complete [ 6186.729749] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:16:27 (1788941787) [ 6194.011673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.614621] LustreError: 193452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.699330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6199.280784] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6199.783997] LustreError: 193453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6208.120620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6208.542773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6213.053822] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6219.611507] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6225.345910] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6232.055357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6232.525539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6239.599637] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6239.703487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6247.380252] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6254.530846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6254.782831] Lustre: lustre-OST0001: new disk, initializing [ 6254.785759] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6254.790638] Lustre: 196293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6254.871904] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6261.625744] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6264.433534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6264.441683] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6264.591442] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6271.457127] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6275.066546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.080512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6275.084363] Lustre: Skipped 1 previous similar message [ 6275.099083] Lustre: Skipped 4 previous similar messages [ 6280.696912] Lustre: server umount lustre-OST0000 complete [ 6282.360215] LustreError: 196754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6282.393620] LustreError: 196754:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6286.746675] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6286.905315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6292.582465] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6292.879816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6300.667248] LustreError: 195275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.686863] LustreError: 195275:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6310.897134] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6310.902379] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.916705] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6310.916713] Lustre: Skipped 3 previous similar messages [ 6310.920119] LustreError: Skipped 2 previous similar messages [ 6310.955660] Lustre: Skipped 1 previous similar message [ 6313.529628] Lustre: server umount lustre-OST0001 complete [ 6321.122512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6321.149256] Lustre: Skipped 1 previous similar message [ 6321.159963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6324.537378] Lustre: server umount lustre-MDT0000 complete [ 6328.535075] LustreError: 193434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941930 with bad export cookie 14847298678829943672 [ 6328.537254] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6328.541889] LustreError: 193434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6328.942228] Lustre: server umount lustre-MDT0001 complete [ 6353.817066] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:19:15 (1788941955) [ 6360.449790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.827559] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6360.884383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6361.169502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.285668] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6375.611238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.725333] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6375.776473] Lustre: 199737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6381.250193] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6390.402962] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6397.254758] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6404.517317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6404.688796] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6412.976267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6413.449728] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6423.023729] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6430.544159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6430.701597] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6431.045769] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6431.048308] Lustre: Skipped 2 previous similar messages [ 6439.461973] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6447.387890] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6453.388728] Lustre: lustre-OST0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 6453.396387] Lustre: Skipped 3 previous similar messages [ 6453.728993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6453.741411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.984250] Lustre: server umount lustre-OST0000 complete [ 6458.487181] LustreError: 202945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.517156] LustreError: 202945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6475.251558] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.268311] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6475.280777] Lustre: Skipped 2 previous similar messages [ 6475.307451] LustreError: Skipped 2 previous similar messages [ 6478.644457] Lustre: server umount lustre-OST0001 complete [ 6483.941753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6485.476747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6485.487059] Lustre: Skipped 7 previous similar messages [ 6497.248442] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6497.630997] Lustre: server umount lustre-MDT0000 complete [ 6501.795575] LustreError: 202620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942103 with bad export cookie 14847298678829945079 [ 6501.798386] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6501.807245] LustreError: 202620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6502.274510] Lustre: server umount lustre-MDT0001 complete [ 6529.826980] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:22:10 (1788942130) [ 6548.208037] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 6559.830494] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 6601.274482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.530924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6601.569777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6601.675760] Lustre: lustre-MDT0000: new disk, initializing [ 6601.802753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6601.843066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6607.347783] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6619.827933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.986396] Lustre: 207829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6620.085280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6620.098525] Lustre: Skipped 1 previous similar message [ 6620.261413] Lustre: lustre-MDT0001: new disk, initializing [ 6620.421894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6620.437259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6625.911469] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6637.209170] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6644.104867] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6651.084677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6651.400315] Lustre: lustre-OST0000: new disk, initializing [ 6651.402700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6651.408317] Lustre: 209801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6659.286037] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6660.137666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6660.152650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6660.332291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6669.543328] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6675.252429] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6675.482326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6680.601449] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6680.857669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6686.179792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6686.183582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6686.189637] LustreError: Skipped 1 previous similar message [ 6686.211231] Lustre: Skipped 2 previous similar messages [ 6686.218391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6686.228578] Lustre: Skipped 4 previous similar messages [ 6689.000610] Lustre: server umount lustre-OST0000 complete [ 6699.070352] Lustre: server umount lustre-MDT0000 complete [ 6701.544044] LustreError: 207840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6701.581389] LustreError: 207840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6703.802392] LustreError: 207823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942305 with bad export cookie 14847298678829946640 [ 6703.809922] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6703.823869] LustreError: 207823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6704.242606] Lustre: server umount lustre-MDT0001 complete [ 6714.100329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6720.505385] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6730.297927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.021455] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6744.208500] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6750.138298] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6757.069546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6763.967659] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6772.144180] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6777.086426] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6777.395258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6782.653777] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6782.955699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6786.025461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6786.028985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.046072] Lustre: Skipped 4 previous similar messages [ 6799.328146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6799.514534] Lustre: server umount lustre-OST0000 complete [ 6806.430598] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6812.309988] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6840.927283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6846.774139] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6851.504997] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6851.518327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6851.519360] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6851.543881] Lustre: Skipped 1 previous similar message [ 6853.818449] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6859.556847] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6866.918147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6866.923281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6866.928571] LustreError: Skipped 1 previous similar message [ 6866.946500] Lustre: Skipped 9 previous similar messages [ 6869.875435] Lustre: server umount lustre-OST0000 complete [ 6879.884842] Lustre: server umount lustre-MDT0000 complete [ 6882.277589] LustreError: 212017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6882.290444] LustreError: 212017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6883.967252] LustreError: 212001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942486 with bad export cookie 14847298678829947515 [ 6883.970236] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6883.979447] LustreError: 212001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6884.291172] Lustre: server umount lustre-MDT0001 complete [ 6892.766450] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:28:13 (1788942493) [ 6899.845270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.389331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.396294] Lustre: Skipped 6 previous similar messages [ 6905.367067] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6915.088771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.640992] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6928.446385] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6933.877657] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6940.665558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.239142] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6955.060141] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6959.535640] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6959.733493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6964.023035] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6964.249481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6971.045277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6971.246605] Lustre: lustre-OST0001: new disk, initializing [ 6971.249785] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6971.254810] Lustre: 221360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6978.195944] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6980.183900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6980.190173] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6980.301641] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6987.978800] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6992.712479] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6992.942184] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6997.860849] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6998.095892] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7001.059511] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.068661] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7001.075681] Lustre: Skipped 4 previous similar messages [ 7001.083401] LustreError: Skipped 1 previous similar message [ 7014.368166] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7014.506739] Lustre: server umount lustre-OST0001 complete [ 7021.680530] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7021.953765] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7027.182416] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7027.487782] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7056.265193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7061.993562] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7062.013185] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7062.022652] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7063.034557] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7070.106753] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7071.644992] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7079.585152] Lustre: server umount lustre-OST0001 complete [ 7103.402893] Lustre: server umount lustre-OST0000 complete [ 7114.292344] Lustre: server umount lustre-MDT0000 complete [ 7117.894162] LustreError: 217969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942719 with bad export cookie 14847298678829948488 [ 7117.899350] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7117.907949] LustreError: 217969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7124.220779] Lustre: server umount lustre-MDT0001 complete [ 7148.223299] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:32:29 (1788942749) [ 7155.650699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.103513] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7156.136263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7161.927739] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7171.255989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.327756] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7171.368356] Lustre: 226604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7176.405958] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7184.356757] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7189.052214] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7198.148234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7198.282440] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7204.914448] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7211.894761] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7221.004961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7221.051076] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7227.464829] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7234.456903] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7235.885098] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7240.410182] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7240.616423] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7244.899556] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7245.144784] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7250.976789] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95e891e2d800.ost_server_uuid 50 [ 7252.462641] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95e891e2d800.ost_server_uuid in IDLE state after 0 sec [ 7253.320472] Lustre: Permanently deactivating lustre-OST0001 [ 7253.349786] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7256.380147] Lustre: Permanently reactivating lustre-OST0001 [ 7260.641617] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7260.651161] LustreError: Skipped 2 previous similar messages [ 7260.656350] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7260.670316] Lustre: Skipped 6 previous similar messages [ 7260.677483] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7260.682711] Lustre: Skipped 21 previous similar messages [ 7265.812413] Lustre: server umount lustre-OST0001 complete [ 7266.276184] LustreError: 230451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7266.288106] LustreError: 230451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 36 previous similar messages [ 7275.327742] Lustre: server umount lustre-OST0000 complete [ 7285.196388] Lustre: server umount lustre-MDT0000 complete [ 7288.616758] LustreError: 226595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942890 with bad export cookie 14847298678829949776 [ 7288.618973] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7288.628774] LustreError: 226595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7288.940989] Lustre: server umount lustre-MDT0001 complete [ 7312.618450] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:35:13 (1788942913) [ 7320.269773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7320.505500] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7320.523849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7320.528759] Lustre: Skipped 5 previous similar messages [ 7325.701250] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7334.337209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.460711] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7334.504967] Lustre: 233298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7339.411301] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7347.118296] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7353.372384] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7360.131584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7360.357909] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7360.374492] Lustre: Permanently deactivating lustre-OST0000 [ 7360.387379] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7365.678952] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7366.711265] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7374.434714] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7381.083536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7381.164256] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7386.979599] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7387.817839] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7394.127582] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7395.685677] Lustre: Permanently reactivating lustre-OST0000 [ 7395.700953] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7395.704721] Lustre: Skipped 2 previous similar messages [ 7397.428486] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7397.431219] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 000000000b5970ef, cur 1788942999 deadline 1788942997 last 1788942967 [ 7397.441442] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7397.464676] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7397.473524] Lustre: Skipped 1 previous similar message [ 7397.505500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7410.656679] Lustre: server umount lustre-OST0001 complete [ 7420.704182] Lustre: server umount lustre-OST0000 complete [ 7431.376135] Lustre: server umount lustre-MDT0000 complete [ 7435.542901] LustreError: 233291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943037 with bad export cookie 14847298678829951428 [ 7435.560282] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7436.184620] Lustre: server umount lustre-MDT0001 complete [ 7448.522234] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 7451.619201] Key type lgssc unregistered [ 7451.995583] LNet: 238176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7452.003787] LNetError: 238176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7452.039491] LNet: Removed LNI 192.168.204.149@tcp [ 7453.014185] Key type .llcrypt unregistered [ 7453.016236] Key type ._llcrypt unregistered [ 7475.669549] Key type ._llcrypt registered [ 7475.672989] Key type .llcrypt registered [ 7475.776923] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 7477.181430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7477.288251] alg: No test for adler32 (adler32-zlib) [ 7478.401894] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 7478.826758] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 7480.544199] Key type lgssc registered [ 7481.599332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7491.581575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7491.604374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7493.050155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.366573] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7503.333723] LustreError: 239639:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.344245] LustreError: 239639:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7505.638740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.746481] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7505.775971] Lustre: 239628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7505.788911] Lustre: Permanently deactivating lustre-MDT0001 [ 7505.813290] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7506.161979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7506.559751] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7506.574483] LustreError: 239670:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7506.596058] LustreError: 239670:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7506.609132] LustreError: 239670:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 7510.600815] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7517.897441] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7523.316382] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7529.569573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7529.907368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7531.947920] LustreError: 241478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7531.959709] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7536.514201] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7537.135666] LustreError: 241478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7538.158571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7543.269883] LustreError: 241606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7543.291697] LustreError: 241606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7543.506037] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7548.389133] LustreError: 241606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7548.409395] LustreError: 241606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7549.796376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.015592] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7555.064326] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7555.068202] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7555.805479] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7562.718879] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7565.344689] Lustre: Permanently reactivating lustre-MDT0001 [ 7565.360712] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7565.367520] Lustre: Skipped 2 previous similar messages [ 7569.291880] Lustre: lustre-MDT0001: Client 53c43e23-3a8c-4249-8663-107560287742 (at 192.168.204.49@tcp) reconnecting [ 7574.993895] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7576.588840] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7582.633333] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7582.894096] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7587.483594] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7587.711283] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7589.977107] Lustre: Permanently deactivating lustre-MDT0001 [ 7589.993639] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7589.998874] Lustre: Skipped 2 previous similar messages [ 7599.169287] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7603.173089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.178177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7603.186535] Lustre: Skipped 3 previous similar messages [ 7603.190286] Lustre: Skipped 2 previous similar messages [ 7608.293549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7608.297993] Lustre: Skipped 1 previous similar message [ 7609.382315] Lustre: server umount lustre-MDT0000 complete [ 7612.896530] LustreError: 239620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943214 with bad export cookie 7426552561054199550 [ 7612.909891] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.323635] Lustre: server umount lustre-MDT0001 complete [ 7622.473901] Lustre: server umount lustre-OST0000 complete [ 7631.062932] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:40:32 (1788943232) [ 7638.940481] Lustre: server umount lustre-OST0001 complete [ 7644.020429] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 7649.945328] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 7684.340812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.520672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7684.534026] Lustre: Skipped 2 previous similar messages [ 7684.558747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7684.687847] Lustre: lustre-MDT0000: new disk, initializing [ 7684.796479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7684.818495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7688.922578] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7697.721185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.877060] Lustre: 247891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7697.951662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7697.957853] Lustre: Skipped 1 previous similar message [ 7698.044411] Lustre: lustre-MDT0001: new disk, initializing [ 7698.227777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7698.278848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7698.288241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7702.314243] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7711.807965] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7717.920816] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7725.328895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7725.523964] Lustre: lustre-OST0000: new disk, initializing [ 7725.526878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7725.534734] Lustre: 249862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7725.605105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7730.956628] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7732.249317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7732.260571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7732.383593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7738.732765] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7740.390098] LustreError: 247897:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7747.348957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7747.504388] Lustre: lustre-OST0001: new disk, initializing [ 7747.512782] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7747.529547] Lustre: 250975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7750.424094] LustreError: 247897:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7750.448355] LustreError: 247899:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7753.431154] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7756.404701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7756.412378] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7756.569405] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7760.528327] LustreError: 247899:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7762.598447] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7764.097315] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7767.010757] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7767.014888] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7767.016876] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7767.023780] LustreError: Skipped 1 previous similar message [ 7767.029291] Lustre: Skipped 3 previous similar messages [ 7771.785622] Lustre: server umount lustre-OST0001 complete [ 7772.138902] LustreError: 249844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.167464] LustreError: 249844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7789.027075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.038241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.059274] Lustre: Skipped 1 previous similar message [ 7789.064830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7792.612722] LustreError: 249847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.614562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.624134] LustreError: 249847:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7794.794899] Lustre: server umount lustre-OST0000 complete [ 7800.802825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7800.815363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.838778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7800.842431] Lustre: Skipped 2 previous similar messages [ 7804.155599] Lustre: server umount lustre-MDT0000 complete [ 7807.764251] LustreError: 248685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943409 with bad export cookie 7426552561054202371 [ 7807.767047] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7807.774633] LustreError: 248685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7822.304312] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7822.622280] Lustre: server umount lustre-MDT0001 complete [ 7836.308535] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 7839.465904] Key type lgssc unregistered [ 7839.691779] LNet: 253008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7839.699862] LNetError: 253008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7839.711790] LNet: Removed LNI 192.168.204.149@tcp [ 7840.656179] Key type .llcrypt unregistered [ 7840.657373] Key type ._llcrypt unregistered [ 7876.947806] Key type ._llcrypt registered [ 7876.951050] Key type .llcrypt registered [ 7877.029489] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 7877.604532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7877.645956] alg: No test for adler32 (adler32-zlib) [ 7878.879185] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 7879.155452] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 7880.889204] Key type lgssc registered [ 7881.836180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7890.242908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7890.258256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7891.658669] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7891.774862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7892.018163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7896.024377] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7904.934083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7905.017757] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7905.051711] Lustre: 255212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7905.334748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7910.359421] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7917.549406] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7922.325726] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7928.214978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.409950] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7928.606356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7934.046553] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7938.749421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7941.288263] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7994.851141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7994.863230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.871581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7995.361300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7995.377644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7995.395486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7995.401405] Lustre: Skipped 1 previous similar message [ 8000.313760] Lustre: server umount lustre-OST0000 complete [ 8003.795960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8016.648428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8017.121246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8020.199433] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8020.210763] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8024.154173] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8032.833503] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8035.555700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8035.582314] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8076.770237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8076.771236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8076.783020] LustreError: Skipped 1 previous similar message [ 8076.786223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8076.794193] Lustre: Skipped 1 previous similar message [ 8079.581616] Lustre: server umount lustre-OST0000 complete [ 8084.452719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8084.460866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.478420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8084.485458] Lustre: Skipped 1 previous similar message [ 8092.130394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8092.138891] Lustre: Skipped 3 previous similar messages [ 8097.248165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8097.486678] Lustre: server umount lustre-MDT0000 complete [ 8100.964431] LustreError: 255203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943703 with bad export cookie 11576968523852680343 [ 8100.970608] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8100.971689] LustreError: 255203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8101.304649] Lustre: server umount lustre-MDT0001 complete [ 8112.212484] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 8115.190759] Key type lgssc unregistered [ 8115.403989] LNet: 260646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8115.409743] LNetError: 260646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8116.454434] LNet: Removed LNI 192.168.204.149@tcp [ 8117.164728] Key type .llcrypt unregistered [ 8117.172375] Key type ._llcrypt unregistered [ 8141.964571] Key type ._llcrypt registered [ 8141.966603] Key type .llcrypt registered [ 8142.070440] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8143.154817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8143.256149] alg: No test for adler32 (adler32-zlib) [ 8144.323659] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 8144.487402] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 8146.193932] Key type lgssc registered [ 8147.509874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8155.878864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8155.920595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8157.270883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8164.603105] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8167.397079] LustreError: 262063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8167.432670] LustreError: 262063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8172.517335] LustreError: 262064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.118023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.545202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8179.225528] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8186.455718] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8192.469229] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8199.015309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8199.316617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8202.427986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8205.515899] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8212.457552] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8240.615159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8240.619426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8240.631564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8240.644849] Lustre: Skipped 1 previous similar message [ 8244.917537] Lustre: server umount lustre-OST0000 complete [ 8250.849487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.861894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8250.866540] Lustre: Skipped 2 previous similar messages [ 8254.404471] Lustre: server umount lustre-MDT0000 complete [ 8255.970727] LustreError: 263956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8255.983495] LustreError: 263956:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8257.232173] LustreError: 262799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943859 with bad export cookie 832469494342781012 [ 8257.242031] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8257.247292] LustreError: 262799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8257.546820] Lustre: server umount lustre-MDT0001 complete [ 8266.996416] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 8269.967217] Key type lgssc unregistered [ 8270.312783] LNet: 266368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8270.318451] LNetError: 266368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8270.353923] LNet: Removed LNI 192.168.204.149@tcp [ 8271.361660] Key type .llcrypt unregistered [ 8271.363460] Key type ._llcrypt unregistered [ 8289.423416] Key type ._llcrypt registered [ 8289.426715] Key type .llcrypt registered [ 8289.590976] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8290.635272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8290.874366] alg: No test for adler32 (adler32-zlib) [ 8291.939197] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 8292.251424] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 8293.960242] Key type lgssc registered [ 8295.141486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8302.306646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8302.321790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.676936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8307.333797] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8313.830150] LustreError: 267586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8313.856046] LustreError: 267586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8314.683673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.955669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8318.720362] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8327.153508] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8332.946411] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8340.182379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8340.606863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8346.127306] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8348.798719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8353.631944] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8369.633435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8369.636663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8369.646348] LustreError: Skipped 1 previous similar message [ 8369.650258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8369.686408] Lustre: Skipped 1 previous similar message [ 8374.756905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.774697] Lustre: Skipped 2 previous similar messages [ 8374.904351] Lustre: server umount lustre-OST0000 complete [ 8379.875549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8379.879891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8379.897222] Lustre: Skipped 1 previous similar message [ 8379.913107] Lustre: Skipped 1 previous similar message [ 8385.002982] LustreError: 267605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8385.025152] Lustre: server umount lustre-MDT0000 complete [ 8385.030618] LustreError: 267605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8389.551784] LustreError: 267566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943991 with bad export cookie 12992910705044430410 [ 8389.564063] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8389.897881] Lustre: server umount lustre-MDT0001 complete [ 8402.267705] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 8405.244462] Key type lgssc unregistered [ 8405.654471] LNet: 271266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8405.661779] LNetError: 271266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8405.677861] LNet: Removed LNI 192.168.204.149@tcp [ 8406.848461] Key type .llcrypt unregistered [ 8406.850777] Key type ._llcrypt unregistered [ 8432.302136] Key type ._llcrypt registered [ 8432.304585] Key type .llcrypt registered [ 8432.438874] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8434.440504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8434.482793] alg: No test for adler32 (adler32-zlib) [ 8435.601287] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 8435.987405] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 8437.720208] Key type lgssc registered [ 8439.266309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8448.819830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8448.862682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.453812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8455.211257] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8460.773168] LustreError: 272684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8460.784878] LustreError: 272684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8463.925762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.355701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8468.041978] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8475.238416] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8481.082601] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8488.348569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8488.804839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8494.981858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8495.835125] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8503.473410] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8530.919119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8530.920697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8530.923961] LustreError: Skipped 1 previous similar message [ 8530.930087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8530.951861] Lustre: Skipped 1 previous similar message [ 8534.044770] Lustre: server umount lustre-OST0000 complete [ 8541.156921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.161482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8541.162446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8541.162452] Lustre: Skipped 1 previous similar message [ 8541.171968] Lustre: Skipped 1 previous similar message [ 8543.960054] Lustre: server umount lustre-MDT0000 complete [ 8546.295779] LustreError: 272684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8547.835111] LustreError: 272664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944149 with bad export cookie 13497120891798920613 [ 8547.838492] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8547.844136] LustreError: 272664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8548.364188] Lustre: server umount lustre-MDT0001 complete [ 8562.035464] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 8565.383674] Key type lgssc unregistered [ 8565.840049] LNet: 276988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8565.858922] LNetError: 276988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8565.887958] LNet: Removed LNI 192.168.204.149@tcp [ 8567.151588] Key type .llcrypt unregistered [ 8567.153533] Key type ._llcrypt unregistered [ 8587.680792] Key type ._llcrypt registered [ 8587.687957] Key type .llcrypt registered [ 8587.948736] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8589.132707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8589.217450] alg: No test for adler32 (adler32-zlib) [ 8590.408309] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 8590.684334] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 8592.384245] Key type lgssc registered [ 8593.709974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8601.796693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8601.826190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.447146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8608.411561] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8613.858156] LustreError: 278223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.881541] LustreError: 278223:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8617.063113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.555989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8622.091859] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8628.247721] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8634.638720] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8644.241963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8645.122304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8648.218346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8653.661501] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8661.359334] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8677.859905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8677.864929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8677.865544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8677.874397] Lustre: Skipped 1 previous similar message [ 8682.978050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8682.987551] Lustre: Skipped 1 previous similar message [ 8688.097462] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8688.108655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.120813] Lustre: Skipped 1 previous similar message [ 8688.569600] Lustre: server umount lustre-OST0000 complete [ 8694.244914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8694.254110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8694.279738] Lustre: Skipped 1 previous similar message [ 8694.287118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8698.337906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8698.339921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8698.386526] Lustre: Skipped 1 previous similar message [ 8700.013418] Lustre: server umount lustre-MDT0000 complete [ 8703.457677] LustreError: 278228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8703.498074] LustreError: 278228:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8704.816065] LustreError: 278188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944306 with bad export cookie 14352969967577595092 [ 8704.831916] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8705.501921] Lustre: server umount lustre-MDT0001 complete [ 8717.299784] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 8720.384862] Key type lgssc unregistered [ 8720.820603] LNet: 281924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8720.827479] LNetError: 281924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8720.858553] LNet: Removed LNI 192.168.204.149@tcp [ 8722.066075] Key type .llcrypt unregistered [ 8722.068220] Key type ._llcrypt unregistered [ 8734.335084] Key type ._llcrypt registered [ 8734.337382] Key type .llcrypt registered [ 8734.456964] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8751.950681] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 8769.137434] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8769.892547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8769.940114] alg: No test for adler32 (adler32-zlib) [ 8771.267450] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 8771.518315] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 8773.265114] Key type lgssc registered [ 8774.818686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8810.974520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8811.010427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.262414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8812.318936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8812.439864] Lustre: lustre-MDT0000: new disk, initializing [ 8812.494702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8812.514677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8817.438794] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8828.157639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.232852] Lustre: 285475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8828.303882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8828.309027] Lustre: Skipped 1 previous similar message [ 8828.396434] Lustre: lustre-MDT0001: new disk, initializing [ 8828.473905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8828.510594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8828.526864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8833.653672] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8843.448811] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8849.787638] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8857.055863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8857.288920] Lustre: lustre-OST0000: new disk, initializing [ 8857.292887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8857.306338] Lustre: 287444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8857.396362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8863.983915] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8865.870159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8865.886772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8866.118989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8873.563241] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8878.362075] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8878.567312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8883.907138] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8884.161113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8886.764144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.769826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.790321] Lustre: Skipped 1 previous similar message [ 8886.791569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8886.819887] Lustre: Skipped 1 previous similar message [ 8891.877733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8891.893398] Lustre: Skipped 1 previous similar message [ 8892.786146] Lustre: server umount lustre-OST0000 complete [ 8896.993177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8896.998829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8897.015803] Lustre: Skipped 1 previous similar message [ 8897.037654] Lustre: Skipped 1 previous similar message [ 8902.116867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8902.123632] Lustre: Skipped 1 previous similar message [ 8907.233214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8907.240353] Lustre: Skipped 1 previous similar message [ 8911.328292] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8911.578106] Lustre: server umount lustre-MDT0000 complete [ 8915.349516] LustreError: 287443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944517 with bad export cookie 8969940846675073494 [ 8915.351388] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8915.359470] LustreError: 287443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8915.751908] Lustre: server umount lustre-MDT0001 complete [ 8924.981104] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:02:06 (1788944526) [ 8931.926462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8932.418142] LustreError: 289868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8932.449215] LustreError: 289868:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8932.503294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8937.629977] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8937.963546] LustreError: 289869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8942.050599] LustreError: 289868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.121963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.514799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8951.037687] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8957.574068] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8962.737465] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8968.874788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8969.205383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8974.561467] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8981.668807] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8991.718318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8991.730982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8991.751394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8993.767874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8993.778889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8996.981405] Lustre: server umount lustre-OST0000 complete [ 9002.985849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9003.004567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9006.859062] Lustre: server umount lustre-MDT0000 complete [ 9009.121842] LustreError: 289869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9009.150847] LustreError: 289869:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9010.133204] LustreError: 289849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944612 with bad export cookie 8969940846675074376 [ 9010.139703] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9010.150089] LustreError: 289849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9010.534893] Lustre: server umount lustre-MDT0001 complete [ 9020.919281] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9023.752332] Key type lgssc unregistered [ 9024.096645] LNet: 293401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9024.109424] LNetError: 293401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9024.129339] LNet: Removed LNI 192.168.204.149@tcp [ 9025.082331] Key type .llcrypt unregistered [ 9025.088490] Key type ._llcrypt unregistered [ 9051.204529] Key type ._llcrypt registered [ 9051.206806] Key type .llcrypt registered [ 9051.297755] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 9066.142727] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9067.211149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9067.243432] alg: No test for adler32 (adler32-zlib) [ 9068.255049] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 9068.485732] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 9070.144246] Key type lgssc registered [ 9071.159956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9110.026387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9110.044657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.220969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9111.245443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9111.324701] Lustre: lustre-MDT0000: new disk, initializing [ 9111.386792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9111.405292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9115.187627] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9124.651286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.745451] Lustre: 297223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9124.784568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9124.786948] Lustre: Skipped 1 previous similar message [ 9124.839200] Lustre: lustre-MDT0001: new disk, initializing [ 9124.897914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9124.924455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9124.929263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9130.482400] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9140.347560] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9145.768311] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9151.609203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9151.835241] Lustre: lustre-OST2710: new disk, initializing [ 9151.838874] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9151.844421] Lustre: 299193:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9151.947026] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9157.329927] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9158.722565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9158.735711] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9158.870427] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9166.590747] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9173.748227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9173.931547] Lustre: lustre-OST03e8: new disk, initializing [ 9173.934907] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9173.940613] Lustre: 300255:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9174.023855] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9180.476351] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9183.349299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9183.362954] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9183.458669] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9187.517642] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9193.502889] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9193.730156] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9198.236894] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9198.409273] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9202.391572] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9202.642713] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9206.279633] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9206.514480] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9211.873372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9211.888257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9211.918451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9212.372618] Lustre: server umount lustre-MDT0000 complete [ 9214.442652] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.718174] LustreError: 297214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944817 with bad export cookie 8839059066826149784 [ 9215.719939] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9215.728609] LustreError: 297214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9215.945594] Lustre: server umount lustre-MDT0001 complete [ 9229.275733] Lustre: server umount lustre-OST2710 complete [ 9243.082823] Lustre: server umount lustre-OST03e8 complete [ 9261.849462] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 9271.117996] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9307.113464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.351501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9307.386372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9307.459435] Lustre: lustre-MDT0000: new disk, initializing [ 9307.505304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9307.526908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9311.192579] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9320.302448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9320.436665] Lustre: 305505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9320.482305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9320.485883] Lustre: Skipped 1 previous similar message [ 9320.542340] Lustre: lustre-MDT0001: new disk, initializing [ 9320.627674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9320.696196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9320.711410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9324.657626] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9332.058570] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9336.831287] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9343.371376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9343.778694] Lustre: lustre-OST0000: new disk, initializing [ 9343.792690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9343.803678] Lustre: 307469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9343.942839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9351.011279] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9351.233733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9351.249708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9351.447098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9360.886136] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9365.182784] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9365.502723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9369.174498] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9369.363406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9372.136615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9372.137914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9372.139505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9372.139726] LustreError: Skipped 1 previous similar message [ 9372.144124] Lustre: Skipped 3 previous similar messages [ 9377.252706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9377.259524] Lustre: Skipped 1 previous similar message [ 9377.407381] Lustre: server umount lustre-OST0000 complete [ 9382.371302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9382.389514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9382.395187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9387.490209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9387.503917] Lustre: Skipped 2 previous similar messages [ 9387.612211] Lustre: server umount lustre-MDT0000 complete [ 9391.056918] LustreError: 305497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944993 with bad export cookie 8839059066826151464 [ 9391.059988] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9391.064553] LustreError: 305497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9391.437638] Lustre: server umount lustre-MDT0001 complete [ 9397.843146] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:09:59 (1788944999) [ 9399.215076] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9400.786420] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:10:02 (1788945002) [ 9404.183049] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9431.351398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.548170] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9431.568857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9431.854798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9436.622360] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9445.575660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.680382] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9445.715763] Lustre: 311332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9450.941242] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9472.434204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9472.697279] Lustre: Denying initial registration attempt from nid 192.168.204.149@tcp, specified as failover [ 9472.704640] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9472.710984] LustreError: 312783:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9472.716503] LustreError: 312783:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9472.720636] LustreError: 312783:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9472.788653] Lustre: server umount lustre-OST0000 complete [ 9472.794478] LustreError: 312783:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9481.699143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9481.707435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9481.723692] Lustre: Skipped 1 previous similar message [ 9481.728603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9485.804865] Lustre: server umount lustre-MDT0000 complete [ 9487.333610] LustreError: 311338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9487.374115] LustreError: 311338:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9489.104549] LustreError: 311324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945091 with bad export cookie 8839059066826152346 [ 9489.112700] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9489.116413] LustreError: 311324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9489.366283] Lustre: server umount lustre-MDT0001 complete [ 9497.777951] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9500.712078] Key type lgssc unregistered [ 9501.020842] LNet: 313906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9501.028673] LNetError: 313906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9501.069329] LNet: Removed LNI 192.168.204.149@tcp [ 9502.150287] Key type .llcrypt unregistered [ 9502.156499] Key type ._llcrypt unregistered [ 9514.862178] Key type ._llcrypt registered [ 9514.865095] Key type .llcrypt registered [ 9515.001957] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9515.912973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9515.927627] alg: No test for adler32 (adler32-zlib) [ 9517.119349] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 9517.331932] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 9518.960477] Key type lgssc registered [ 9520.036477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9568.853038] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9577.402698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9577.435602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9578.690795] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9578.704788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9578.825672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9583.428286] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9592.223990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9592.362601] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9592.414660] Lustre: 316669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9592.685255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9597.429472] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9605.149374] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9610.770748] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9630.911433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9631.149288] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9631.335899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9636.986256] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9643.442946] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9649.121909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9649.134193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9649.146213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9649.151449] Lustre: Skipped 1 previous similar message [ 9651.411402] Lustre: server umount lustre-OST0000 complete [ 9659.366512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9659.377766] LustreError: Skipped 1 previous similar message [ 9659.378807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.389111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.389119] Lustre: Skipped 1 previous similar message [ 9659.414510] Lustre: Skipped 1 previous similar message [ 9664.489059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9664.504152] Lustre: Skipped 2 previous similar messages [ 9669.088781] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9669.469767] Lustre: server umount lustre-MDT0000 complete [ 9673.506047] LustreError: 316659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945275 with bad export cookie 2485215477039499725 [ 9673.512799] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9673.517199] LustreError: 316659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9673.943939] Lustre: server umount lustre-MDT0001 complete [ 9685.114867] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9687.911554] Key type lgssc unregistered [ 9688.229238] LNet: 320242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9688.236614] LNetError: 320242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9688.274608] LNet: Removed LNI 192.168.204.149@tcp [ 9689.216073] Key type .llcrypt unregistered [ 9689.218572] Key type ._llcrypt unregistered [ 9713.207296] Key type ._llcrypt registered [ 9713.210897] Key type .llcrypt registered [ 9713.304974] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9714.391906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9714.428595] alg: No test for adler32 (adler32-zlib) [ 9715.458394] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 9715.682725] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 9717.344370] Key type lgssc registered [ 9718.673264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9726.572435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9726.591133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9728.053064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9732.272945] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9738.221068] LustreError: 321655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.248365] LustreError: 321655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9741.155355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9741.617378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9746.653856] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9754.544501] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9759.964446] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9767.725960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9768.111460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9773.799080] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9780.445463] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9797.092082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9797.103603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9797.119722] Lustre: Skipped 2 previous similar messages [ 9797.132726] Lustre: Skipped 2 previous similar messages [ 9798.809572] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 9802.214444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9802.226531] Lustre: Skipped 1 previous similar message [ 9802.931639] Lustre: server umount lustre-MDT0000 complete [ 9803.885671] LustreError: 321651:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9806.709478] LustreError: 321635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945408 with bad export cookie 10415382605070112123 [ 9806.714168] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9806.720718] LustreError: 321635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9807.329249] LustreError: 321652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9807.329605] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9807.366770] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9811.369808] LustreError: 321650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9811.370071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9811.370081] Lustre: Skipped 1 previous similar message [ 9811.418377] LustreError: 321650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9816.483226] LustreError: 321650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9816.505435] LustreError: 321650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9820.335079] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 9820.342946] Lustre: Skipped 3 previous similar messages [ 9821.152710] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9821.427686] Lustre: server umount lustre-MDT0001 complete [ 9827.695192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9835.352592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.361067] LustreError: 320963:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f591b49c x1875845124764416/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9851.637333] LustreError: 323488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.654726] LustreError: 323488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9851.700395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9851.746733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9855.481504] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9862.920245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9863.160807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9866.177757] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9871.340624] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9871.516400] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9877.348310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 9883.401191] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9884.717531] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9891.821535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9891.834227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9891.840487] Lustre: Skipped 1 previous similar message [ 9895.569756] Lustre: server umount lustre-OST0000 complete [ 9902.062095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.085467] Lustre: Skipped 2 previous similar messages [ 9905.697197] Lustre: server umount lustre-MDT0000 complete [ 9907.171890] LustreError: 325285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9907.191474] LustreError: 325285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9908.631706] LustreError: 322391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945510 with bad export cookie 10415382605070119963 [ 9908.633357] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9908.650162] LustreError: 322391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9908.936121] Lustre: server umount lustre-MDT0001 complete [ 9919.851705] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9922.554613] Key type lgssc unregistered [ 9922.844557] LNet: 327953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9922.855799] LNetError: 327953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9923.878066] LNet: Removed LNI 192.168.204.149@tcp [ 9924.837167] Key type .llcrypt unregistered [ 9924.850299] Key type ._llcrypt unregistered [ 9946.527469] Key type ._llcrypt registered [ 9946.529748] Key type .llcrypt registered [ 9946.616130] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9947.566849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9947.653431] alg: No test for adler32 (adler32-zlib) [ 9948.685784] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 9948.879594] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 9950.544225] Key type lgssc registered [ 9951.451200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9959.064295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9959.085484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.640691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.247745] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9970.146137] LustreError: 329358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.159806] LustreError: 329358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9972.858977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.251282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9976.837421] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9982.816978] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9988.519605] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9993.697650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9993.701994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9993.710151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.751621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9995.753683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.784177] Lustre: Skipped 1 previous similar message [ 9997.824198] Lustre: server umount lustre-MDT0000 complete [ 9998.884778] LustreError: 330750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.920939] LustreError: 330750:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10001.294970] LustreError: 329338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945603 with bad export cookie 18162026729661721686 [10001.296098] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10001.308620] LustreError: 329338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10001.652816] Lustre: server umount lustre-MDT0001 complete [10014.957298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.306068] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10015.347592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10015.648842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10019.437547] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10027.675569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.806792] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10027.860836] Lustre: 331945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10028.140738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10033.560826] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10040.597333] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10046.235171] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10054.697621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10054.924341] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10054.932349] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10054.940312] LustreError: 333829:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10054.945438] LustreError: 333829:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10054.954059] LustreError: 333829:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10055.062739] Lustre: server umount lustre-OST0000 complete [10055.066604] LustreError: 333829:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10064.567442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10064.737771] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10064.918314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10069.510489] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10073.737041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10075.476208] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10081.679717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10081.783171] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10081.855386] Lustre: lustre-OST0001: new disk, initializing [10081.857928] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10081.863749] Lustre: 335370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10081.924946] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10086.366501] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10090.082402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10090.096562] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10090.211630] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10091.599438] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10095.592186] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10095.592336] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10095.600351] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10095.603661] LustreError: Skipped 1 previous similar message [10095.612573] Lustre: Skipped 1 previous similar message [10099.757213] Lustre: server umount lustre-OST0001 complete [10100.713027] LustreError: 334295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10100.725543] LustreError: 334295:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10104.289709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10104.296339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10104.302630] Lustre: Skipped 1 previous similar message [10104.306572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10105.829232] LustreError: 335380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10105.846749] LustreError: 335380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10110.951276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.955262] LustreError: 334295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.959995] Lustre: Skipped 4 previous similar messages [10117.088202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10117.217894] Lustre: server umount lustre-OST0000 complete [10121.186562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10121.203488] Lustre: Skipped 1 previous similar message [10121.216185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10121.225337] Lustre: Skipped 3 previous similar messages [10126.518058] Lustre: server umount lustre-MDT0000 complete [10130.309526] LustreError: 331939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945732 with bad export cookie 18162026729661722239 [10130.322365] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10130.335996] LustreError: 331939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10130.722099] Lustre: server umount lustre-MDT0001 complete [10142.753247] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [10146.057155] Key type lgssc unregistered [10146.393367] LNet: 337329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10146.404868] LNetError: 337329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10146.424291] LNet: Removed LNI 192.168.204.149@tcp [10147.811282] Key type .llcrypt unregistered [10147.813064] Key type ._llcrypt unregistered [10212.700606] Key type ._llcrypt registered [10212.704821] Key type .llcrypt registered [10212.802881] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [10225.372957] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [10226.551528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10226.601088] alg: No test for adler32 (adler32-zlib) [10227.753293] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [10228.066478] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [10229.777446] Key type lgssc registered [10230.791198] Lustre: Echo OBD driver; http://www.lustre.org/ [10263.168385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10263.204411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.518290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10264.562959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10264.692359] Lustre: lustre-MDT0000: new disk, initializing [10264.818637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10264.862178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10269.308049] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10279.756264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.900039] Lustre: 342237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10279.956233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10279.962774] Lustre: Skipped 1 previous similar message [10280.045152] Lustre: lustre-MDT0001: new disk, initializing [10280.195438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10280.285616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10280.302030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10284.494265] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10293.170876] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10298.469640] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10304.625727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10304.822252] Lustre: lustre-OST0000: new disk, initializing [10304.825773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10304.833071] Lustre: 344207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10304.912548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10310.856586] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10313.820370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10313.830624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10313.934908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10320.332706] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10325.797781] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10326.016744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10330.760700] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10331.089907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10334.691936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10334.692794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10334.702887] LustreError: Skipped 1 previous similar message [10334.734458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10339.337664] Lustre: server umount lustre-OST0000 complete [10344.931428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10344.947883] Lustre: Skipped 1 previous similar message [10344.952901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10344.956787] Lustre: Skipped 1 previous similar message [10348.988139] Lustre: server umount lustre-MDT0000 complete [10352.083490] LustreError: 342232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945954 with bad export cookie 11616198032703430704 [10352.092716] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10352.097494] LustreError: 342232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10352.452220] Lustre: server umount lustre-MDT0001 complete [10359.670632] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:26:01 (1788945961) [10365.331180] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:26:06 (1788945966) [10372.399723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10372.907444] LustreError: 346962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10372.935783] LustreError: 346962:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10372.980860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10377.205176] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10378.210681] LustreError: 346963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10383.344329] LustreError: 346962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10387.125686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10387.608544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10393.518928] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10401.915195] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10406.981628] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10412.855709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10413.131975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10419.488619] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10427.986065] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10440.431505] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10442.442262] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10444.556243] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10446.371053] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10448.391875] Lustre: Failing over lustre-MDT0000 [10448.773906] Lustre: server umount lustre-MDT0000 complete [10448.892587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10448.903951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10448.928376] Lustre: Skipped 1 previous similar message [10448.944953] LustreError: 346963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10453.477050] LustreError: 346962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10453.478385] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10453.507103] LustreError: 346962:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10463.716836] LustreError: 346958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10463.716836] LustreError: 346959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10463.716846] LustreError: 346959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10463.753113] LustreError: 346958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10467.169877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10467.290669] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10467.357190] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (not set up) [10467.511011] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10467.553094] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10472.141819] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10472.567203] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10472.954565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10473.000834] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10473.074184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10482.037470] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10483.493719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10486.856991] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10488.755816] Lustre: Failing over lustre-MDT0000 [10489.074785] Lustre: server umount lustre-MDT0000 complete [10493.066949] LustreError: 346957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.085529] LustreError: 346957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10493.412022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10493.420922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10493.440898] Lustre: Skipped 1 previous similar message [10507.486762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10507.672516] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10507.946901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10507.998203] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10508.399842] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10512.013471] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10513.396893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10513.407139] Lustre: Skipped 2 previous similar messages [10513.434619] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10513.473425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10519.495777] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10520.837700] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10528.743190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10528.745624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10528.756640] Lustre: Skipped 2 previous similar messages [10528.763504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10528.783177] Lustre: Skipped 2 previous similar messages [10531.821138] Lustre: server umount lustre-MDT0000 complete [10533.859675] LustreError: 348504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10533.891134] LustreError: 348504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [10535.364259] LustreError: 346987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946137 with bad export cookie 11616198032703433413 [10535.366863] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10535.378877] LustreError: 346987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10535.882041] Lustre: server umount lustre-MDT0001 complete [10549.226955] Lustre: server umount lustre-OST0000 complete [10563.862057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10569.041389] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10577.686566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10578.160118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10578.168833] Lustre: Skipped 1 previous similar message [10582.500509] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10588.938628] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10593.498629] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10601.329656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10609.205065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10609.400604] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10618.666624] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10623.464588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10623.471332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.498646] Lustre: Skipped 2 previous similar messages [10623.501110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10625.196355] Lustre: server umount lustre-OST0000 complete [10634.724626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10634.728268] Lustre: Skipped 5 previous similar messages [10635.527884] Lustre: server umount lustre-MDT0000 complete [10638.775971] LustreError: 353300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946240 with bad export cookie 11616198032703434162 [10638.786929] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10639.142482] Lustre: server umount lustre-MDT0001 complete [10649.269797] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [10652.193953] Key type lgssc unregistered [10652.484917] LNet: 356850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10652.492341] LNetError: 356850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10652.511669] LNet: Removed LNI 192.168.204.149@tcp [10653.479932] Key type .llcrypt unregistered [10653.483377] Key type ._llcrypt unregistered [10680.351804] Key type ._llcrypt registered [10680.356339] Key type .llcrypt registered [10680.550609] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [10681.948122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10682.053772] alg: No test for adler32 (adler32-zlib) [10683.364480] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [10683.640413] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [10685.329434] Key type lgssc registered [10686.415848] Lustre: Echo OBD driver; http://www.lustre.org/ [10695.301908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10695.318289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10696.869653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10701.282841] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10706.405598] LustreError: 358364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.426124] LustreError: 358364:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10710.101224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10710.578381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10715.458410] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10723.071110] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10727.740668] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10734.884657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10735.014235] LustreError: 360184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10735.363240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10741.751529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10743.651332] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10752.423500] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10762.211621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10762.221413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10762.226038] Lustre: Skipped 2 previous similar messages [10762.237031] Lustre: Skipped 2 previous similar messages [10764.332100] Lustre: server umount lustre-MDT0000 complete [10767.330908] LustreError: 358361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.356041] LustreError: 358361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10768.114044] LustreError: 358345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946370 with bad export cookie 1065319196562372228 [10768.129832] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10768.133217] LustreError: 358345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10768.390801] Lustre: server umount lustre-MDT0001 complete [10781.139748] Lustre: server umount lustre-OST0000 complete [10796.548986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.084563] LustreError: 362076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.178183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10802.180522] LustreError: 362077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.652289] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10812.326400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10812.829716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10817.058292] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10824.614973] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10830.632253] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10837.132259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10837.541504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10843.525933] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10843.667922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10851.331993] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10864.110920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.117580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10864.118281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10864.132036] Lustre: Skipped 2 previous similar messages [10869.220127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10869.233157] Lustre: Skipped 4 previous similar messages [10870.318866] Lustre: server umount lustre-MDT0000 complete [10874.346100] LustreError: 362073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.367701] LustreError: 362073:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10874.662973] LustreError: 362058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946476 with bad export cookie 1065319196562373334 [10874.672306] LustreError: 362058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10874.673684] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10875.095811] Lustre: server umount lustre-MDT0001 complete [10889.393156] Lustre: server umount lustre-OST0000 complete [10905.800055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10906.489716] LustreError: 365987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10906.652066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10912.397603] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10920.711567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10924.693647] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10932.926421] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10938.420387] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10945.499872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.986420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10945.993394] Lustre: Skipped 1 previous similar message [10951.126996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10952.138922] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [10959.646663] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10966.122839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10966.133032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10967.325332] Lustre: server umount lustre-OST0000 complete [10971.625179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10971.633438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10971.642417] Lustre: Skipped 1 previous similar message [10971.657896] Lustre: Skipped 2 previous similar messages [10977.770513] Lustre: server umount lustre-MDT0000 complete [10981.549687] LustreError: 365969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946583 with bad export cookie 1065319196562374363 [10981.576942] LustreError: 365969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10981.582311] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10982.043690] Lustre: server umount lustre-MDT0001 complete [10994.918716] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [10998.248351] Key type lgssc unregistered [10998.618735] LNet: 369517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10998.627037] LNetError: 369517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10998.650676] LNet: Removed LNI 192.168.204.149@tcp [10999.569168] Key type .llcrypt unregistered [10999.572659] Key type ._llcrypt unregistered [11028.514926] Key type ._llcrypt registered [11028.519879] Key type .llcrypt registered [11028.667754] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [11030.475878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11030.552720] alg: No test for adler32 (adler32-zlib) [11031.693298] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [11032.264228] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [11034.029346] Key type lgssc registered [11035.877260] Lustre: Echo OBD driver; http://www.lustre.org/ [11044.367868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11044.393293] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11044.397681] LustreError: 370956:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11044.407960] LustreError: 370956:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11044.414734] LustreError: 370956:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11044.421986] LustreError: 370956:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11044.432201] LustreError: 370956:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11052.750579] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11052.754921] LustreError: 371353:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11052.789489] LustreError: 371353:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11052.803420] LustreError: 371353:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11052.813180] LustreError: 371353:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11052.826150] LustreError: 371353:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11072.180371] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [11074.580638] Key type lgssc unregistered [11075.011488] LNet: 372358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11075.019644] LNetError: 372358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11075.036220] LNet: Removed LNI 192.168.204.149@tcp [11075.557925] Key type .llcrypt unregistered [11075.564148] Key type ._llcrypt unregistered [11093.011602] Key type ._llcrypt registered [11093.020287] Key type .llcrypt registered [11093.205346] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [11109.548832] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [11110.737628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11110.790364] alg: No test for adler32 (adler32-zlib) [11111.881756] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [11112.144917] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [11113.851239] Key type lgssc registered [11114.977154] Lustre: Echo OBD driver; http://www.lustre.org/ [11150.655966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11150.676920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11151.935502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11151.995556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11152.138538] Lustre: lustre-MDT0000: new disk, initializing [11152.297634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11152.323171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11159.067927] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11170.829085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.083798] Lustre: 375513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11171.201105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11171.204381] Lustre: Skipped 1 previous similar message [11171.343566] Lustre: lustre-MDT0001: new disk, initializing [11171.544364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11171.659468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11171.688063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11176.469816] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11184.534507] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11189.822469] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11196.110321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11196.503789] Lustre: lustre-OST0000: new disk, initializing [11196.510732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11196.520366] Lustre: 377479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11196.701816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11203.027813] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11205.174466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11205.194706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11205.288896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11213.097596] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11218.636240] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11218.836752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11224.129941] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11224.435104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11231.208390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11231.210511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11231.211188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11231.246745] LustreError: Skipped 1 previous similar message [11232.664555] Lustre: server umount lustre-OST0000 complete [11238.383105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11238.396572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11238.408485] Lustre: Skipped 1 previous similar message [11238.415915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11238.425120] Lustre: Skipped 1 previous similar message [11241.447250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11241.450711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11243.403591] Lustre: server umount lustre-MDT0000 complete [11246.472095] LustreError: 375503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946848 with bad export cookie 8899646580182618764 [11246.474759] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11246.480363] LustreError: 375503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11246.656646] Lustre: server umount lustre-MDT0001 complete [11255.487136] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11257.358225] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:40:58 (1788946858) [11263.760696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11264.124812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11269.128166] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11269.605483] LustreError: 379939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11269.622909] LustreError: 379939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11274.731253] LustreError: 379940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11277.939938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11278.477089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11283.052196] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11291.100144] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11297.042770] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11303.196666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11303.331405] LustreError: 381760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.468936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11309.884223] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11317.821907] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11324.753635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11324.953430] Lustre: lustre-OST0001: new disk, initializing [11324.956401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11324.962208] Lustre: 382774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11325.065382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11330.652804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11330.671994] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11330.768731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11331.738263] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11341.484587] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11346.401960] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11346.403685] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11346.410507] LustreError: Skipped 1 previous similar message [11346.416442] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11346.436402] Lustre: Skipped 1 previous similar message [11346.463461] Lustre: Skipped 2 previous similar messages [11351.349778] Lustre: server umount lustre-OST0001 complete [11351.529205] LustreError: 381762:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11356.643171] LustreError: 381761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11356.657517] LustreError: 381761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11366.895920] LustreError: 381762:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.908239] LustreError: 381762:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11369.953610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11369.966505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11369.976181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11369.979660] Lustre: Skipped 1 previous similar message [11374.741737] Lustre: server umount lustre-OST0000 complete [11380.201445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11380.228366] Lustre: Skipped 1 previous similar message [11380.233492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11380.237602] Lustre: Skipped 4 previous similar messages [11385.078863] Lustre: server umount lustre-MDT0000 complete [11385.314776] LustreError: 383702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11385.342046] LustreError: 383702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11389.642317] LustreError: 379920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946991 with bad export cookie 8899646580182619646 [11389.653996] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11389.656099] LustreError: 379920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11396.284463] Lustre: server umount lustre-MDT0001 complete [11406.955636] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [11410.886315] Key type lgssc unregistered [11411.350782] LNet: 384765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11411.370907] LNetError: 384765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11411.433314] LNet: Removed LNI 192.168.204.149@tcp [11412.712375] Key type .llcrypt unregistered [11412.714260] Key type ._llcrypt unregistered [11463.756498] Key type ._llcrypt registered [11463.758726] Key type .llcrypt registered [11463.873317] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [11464.800046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11464.810499] alg: No test for adler32 (adler32-zlib) [11465.804279] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [11465.997493] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [11467.648224] Key type lgssc registered [11468.918829] Lustre: Echo OBD driver; http://www.lustre.org/ [11478.749069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11478.764415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11480.071569] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11480.091195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11480.243310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11487.028565] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11496.802852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11496.895981] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11496.941280] Lustre: 387564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11497.199883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11501.084430] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11509.193382] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11514.802406] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11517.922047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11517.926586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11517.951358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11521.006271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11521.008250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.590382] Lustre: server umount lustre-MDT0000 complete [11529.601155] LustreError: 387556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947131 with bad export cookie 9982819908528064031 [11529.604330] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11529.624138] LustreError: 387556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11529.921696] Lustre: server umount lustre-MDT0001 complete [11540.408439] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:45:41 (1788947141) [11546.340941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11546.779298] LustreError: 390072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11546.792370] LustreError: 390072:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11546.840216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11550.313873] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11552.227118] LustreError: 390073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.357818] LustreError: 390072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.864462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11558.286199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11562.488724] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11570.150873] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11577.101289] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11584.951439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11585.176727] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11585.350457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11591.973521] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11601.265720] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11611.633108] Lustre: Permanently deactivating lustre-OST0000 [11611.661575] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11620.915550] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11622.036832] LustreError: 393061:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11622.047145] LustreError: 393061:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11622.053667] LustreError: 393061:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11625.865891] LustreError: 393109:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11625.871471] LustreError: 393109:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11634.262543] Lustre: server umount lustre-OST0000 complete [11637.333690] LustreError: 393360:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11637.338967] LustreError: 393360:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11640.293409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11640.306430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11640.341095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11640.358771] Lustre: Skipped 1 previous similar message [11643.873236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11643.878496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11648.999179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11649.009049] Lustre: Skipped 2 previous similar messages [11654.117485] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11654.316300] Lustre: server umount lustre-MDT0000 complete [11658.552627] LustreError: 392765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947260 with bad export cookie 9982819908528064612 [11658.554639] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11658.558567] LustreError: 392765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11668.149470] Lustre: server umount lustre-MDT0001 complete [11678.373270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.907185] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11687.515760] LustreError: 394521:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11694.035737] Lustre: 394860:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.149@tcp [11696.369426] Lustre: server umount MGS complete [11707.804552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11708.283555] LustreError: 395584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11708.308247] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11708.312817] Lustre: Skipped 1 previous similar message [11708.361960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11713.001273] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11713.511672] LustreError: 395585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11721.162145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11721.539503] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11726.533868] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11735.124641] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11740.361847] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11742.010054] Lustre: Permanently reactivating lustre-OST0000 [11742.036970] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11742.042256] Lustre: Skipped 2 previous similar messages [11770.336729] LustreError: 397365:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11778.597942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11779.066943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11779.079135] Lustre: Skipped 1 previous similar message [11787.141610] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11795.668289] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11804.642912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11804.654197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11804.669785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11804.684540] Lustre: Skipped 2 previous similar messages [11808.342372] Lustre: server umount lustre-OST0000 complete [11813.864735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11813.872077] LustreError: Skipped 1 previous similar message [11813.873851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11813.893320] Lustre: Skipped 1 previous similar message [11819.425560] Lustre: server umount lustre-MDT0000 complete [11821.026456] LustreError: 395584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11821.050188] LustreError: 395584:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11823.915048] LustreError: 395564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947425 with bad export cookie 9982819908528065662 [11823.916974] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11823.924778] LustreError: 395564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11824.506933] Lustre: server umount lustre-MDT0001 complete [11838.738654] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [11842.377339] Key type lgssc unregistered [11842.822194] LNet: 399950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11842.833692] LNetError: 399950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11842.872696] LNet: Removed LNI 192.168.204.149@tcp [11843.715756] Key type .llcrypt unregistered [11843.725671] Key type ._llcrypt unregistered [11864.475788] Key type ._llcrypt registered [11864.478482] Key type .llcrypt registered [11864.597396] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [11882.115282] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [11883.185882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11883.246868] alg: No test for adler32 (adler32-zlib) [11884.506664] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [11884.807748] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [11886.528507] Key type lgssc registered [11888.007729] Lustre: Echo OBD driver; http://www.lustre.org/ [11933.024530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11933.047883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11934.448195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11934.511560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11934.653413] Lustre: lustre-MDT0000: new disk, initializing [11934.808411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11934.830260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11941.049610] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11953.730391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11953.948543] Lustre: 403104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11954.043499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11954.047733] Lustre: Skipped 1 previous similar message [11954.188717] Lustre: lustre-MDT0001: new disk, initializing [11954.313309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11954.377343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11954.387669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11959.979502] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11972.495397] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11980.024235] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11988.645945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11989.030443] Lustre: lustre-OST0000: new disk, initializing [11989.034630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11989.042707] Lustre: 405073:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11989.159372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11996.727651] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [11999.352218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11999.367589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11999.501489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12008.312976] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12014.228687] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12014.511818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12020.442751] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12020.675538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12025.315359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12025.319249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12025.334828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12029.485770] Lustre: server umount lustre-OST0000 complete [12035.553456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12035.565251] Lustre: Skipped 2 previous similar messages [12035.566509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12035.578182] Lustre: Skipped 1 previous similar message [12040.206321] Lustre: server umount lustre-MDT0000 complete [12044.811972] LustreError: 403095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947646 with bad export cookie 9855064742344840296 [12044.813160] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12044.822676] LustreError: 403095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12045.146843] Lustre: server umount lustre-MDT0001 complete [12054.465746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12056.885947] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:54:17 (1788947657) [12063.961306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12064.537175] LustreError: 407535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12064.545740] LustreError: 407535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12064.633700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12069.860400] LustreError: 407536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12070.035333] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12073.958438] LustreError: 407535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12079.075788] LustreError: 407536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12079.706995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12080.346730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12085.826945] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12095.725675] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12102.122911] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12110.311165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12110.690662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12117.672280] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12125.876614] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12127.746108] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12134.855339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12144.617885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12144.626430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.640021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12144.659373] Lustre: Skipped 1 previous similar message [12146.825950] Lustre: server umount lustre-OST0000 complete [12151.780148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12151.808890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.827865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12151.839033] Lustre: Skipped 1 previous similar message [12157.379578] Lustre: server umount lustre-MDT0000 complete [12159.969450] LustreError: 407536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12159.987250] LustreError: 407536:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12161.703752] LustreError: 407516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947763 with bad export cookie 9855064742344841171 [12161.712353] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12161.717254] LustreError: 407516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12162.177610] Lustre: server umount lustre-MDT0001 complete [12174.730203] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [12178.354687] Key type lgssc unregistered [12178.766746] LNet: 411263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12178.802732] LNetError: 411263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12178.844728] LNet: Removed LNI 192.168.204.149@tcp [12179.938256] Key type .llcrypt unregistered [12179.940078] Key type ._llcrypt unregistered [12210.191158] Key type ._llcrypt registered [12210.192800] Key type .llcrypt registered [12210.323634] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [12210.681546] Key type .llcrypt unregistered [12210.683291] Key type ._llcrypt unregistered [12230.300288] Key type ._llcrypt registered [12230.302599] Key type .llcrypt registered [12230.494355] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [12231.876169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12231.916631] alg: No test for adler32 (adler32-zlib) [12233.017769] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [12233.333050] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [12235.048581] Key type lgssc registered [12236.297856] Lustre: Echo OBD driver; http://www.lustre.org/ [12245.363728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12245.388552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12246.907294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12253.500446] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12256.236081] LustreError: 413469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.269929] LustreError: 413469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12261.353986] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12265.137409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12265.586678] LustreError: 413469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12265.630147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12271.089240] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12271.126587] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12272.201340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12273.634751] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12277.220189] LustreError: 413469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.237360] LustreError: 413469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12282.238673] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12287.463266] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12287.495303] LustreError: 413468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12290.031034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12290.508794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12296.400803] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12305.894823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12305.897545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12305.909921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12305.954210] Lustre: Skipped 1 previous similar message [12311.016597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12311.035274] Lustre: Skipped 2 previous similar messages [12311.996405] Lustre: server umount lustre-OST0000 complete [12316.129887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12316.140825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12316.143167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12321.250661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12321.264630] Lustre: Skipped 2 previous similar messages [12322.249786] Lustre: server umount lustre-MDT0000 complete [12326.373861] LustreError: 413465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.386975] LustreError: 413465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12326.613023] LustreError: 413450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947928 with bad export cookie 12227260741675556089 [12326.619811] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12326.626246] LustreError: 413450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12327.003913] Lustre: server umount lustre-MDT0001 complete [12338.235308] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [12341.150992] Key type lgssc unregistered [12341.422838] LNet: 416617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12341.429833] LNetError: 416617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12342.503257] LNet: Removed LNI 192.168.204.149@tcp [12343.460950] Key type .llcrypt unregistered [12343.462804] Key type ._llcrypt unregistered [12372.586446] Key type ._llcrypt registered [12372.590148] Key type .llcrypt registered [12372.760589] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [12374.464102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12374.516464] alg: No test for adler32 (adler32-zlib) [12375.775577] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [12376.162356] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [12377.848311] Key type lgssc registered [12379.695656] Lustre: Echo OBD driver; http://www.lustre.org/ [12388.386447] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12388.410606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12404.064548] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12409.184297] LustreError: 418011:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.149@tcp: failed processing log, type 4: rc = -110 [12435.937483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12445.521246] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12455.707814] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12463.493290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12464.130056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12464.215728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12468.840663] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12469.229740] LustreError: 418036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.243454] LustreError: 418036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12471.270462] LustreError: 419059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12471.287421] LustreError: 419059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12476.390706] LustreError: 419058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12476.422908] LustreError: 419058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12478.656564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12479.104068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12484.122585] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12492.024531] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12498.441155] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12508.120275] Lustre: server umount lustre-OST0000 complete [12513.767959] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12513.770442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12513.796735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12515.300534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12515.303564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12520.422379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12520.430149] Lustre: Skipped 2 previous similar messages [12523.570873] Lustre: server umount lustre-MDT0000 complete [12525.539412] Lustre: 417349:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788948111/real 1788948111] req@00000000907645c8 x1875847915011328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788948127 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12525.548600] LustreError: 419064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12525.574278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12525.585168] LustreError: 419064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12526.565764] Lustre: 417346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788948112/real 1788948112] req@0000000010c7e08e x1875847915012096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788948128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12527.694616] LustreError: 420554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948129 with bad export cookie 18075326994300167498 [12527.705581] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12527.707907] LustreError: 420554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12528.005235] Lustre: server umount lustre-MDT0001 complete [12540.923302] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [12543.984325] Key type lgssc unregistered [12544.332398] LNet: 421593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12544.348623] LNetError: 421593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12544.369395] LNet: Removed LNI 192.168.204.149@tcp [12545.447780] Key type .llcrypt unregistered [12545.450927] Key type ._llcrypt unregistered [12572.452324] Key type ._llcrypt registered [12572.458331] Key type .llcrypt registered [12572.619292] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [12573.580326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12573.604716] alg: No test for adler32 (adler32-zlib) [12574.812799] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [12575.214902] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [12576.952686] Key type lgssc registered [12578.230894] Lustre: Echo OBD driver; http://www.lustre.org/ [12587.316924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12587.341276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12588.961104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12593.491389] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12598.258877] LustreError: 423008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12598.272850] LustreError: 423008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12603.073383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12603.437528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12608.912311] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12617.427407] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12624.505524] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12633.204921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12633.673081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12641.198340] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12649.565466] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12658.219538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12664.803337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12664.819146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12664.851853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12666.520911] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [12668.390996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12668.393349] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12668.414842] Lustre: Skipped 1 previous similar message [12670.771621] Lustre: server umount lustre-MDT0000 complete [12671.597564] LustreError: 425607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12673.422190] LustreError: 422989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.49@tcp arrived at 1788948275 with bad export cookie 12651578000036762366 [12673.436607] LustreError: 422989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12673.512149] LustreError: 425607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12673.530580] LustreError: 425607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12675.250794] LustreError: 422988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948277 with bad export cookie 12651578000036761701 [12675.266325] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12678.626556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12678.626561] LustreError: 423003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12678.662963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12678.676500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12678.680500] Lustre: Skipped 2 previous similar messages [12681.473454] Lustre: server umount lustre-OST0000 complete [12685.795672] LustreError: 423007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12685.810880] LustreError: 423007:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12688.875486] Lustre: server umount lustre-MDT0001 complete [12700.978645] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [12703.467819] Key type lgssc unregistered [12703.763377] LNet: 426713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12703.770266] LNetError: 426713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12703.784788] LNet: Removed LNI 192.168.204.149@tcp [12704.639195] Key type .llcrypt unregistered [12704.644174] Key type ._llcrypt unregistered [12727.687764] Key type ._llcrypt registered [12727.689410] Key type .llcrypt registered [12727.789449] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [12729.141444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12729.193312] alg: No test for adler32 (adler32-zlib) [12730.263660] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [12730.497374] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [12732.144232] Key type lgssc registered [12733.309212] Lustre: Echo OBD driver; http://www.lustre.org/ [12741.557600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12741.576622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12742.963039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12747.776750] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12752.358251] LustreError: 428127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12752.366166] LustreError: 428127:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12756.377926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12756.778070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12761.693909] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12769.058319] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12774.496253] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12781.206331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12781.559696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12787.705985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12788.004494] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12795.814596] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12802.701484] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [12803.043329] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12803.052947] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12805.805476] Lustre: server umount lustre-MDT0001 complete [12807.788730] LustreError: 428123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12813.282701] LustreError: 428127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12813.282760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12813.305920] LustreError: 428127:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12813.318785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12813.327304] Lustre: Skipped 1 previous similar message [12813.331037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12813.337224] Lustre: Skipped 2 previous similar messages [12818.402086] LustreError: 428128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12818.403369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12818.805690] Lustre: server umount lustre-OST0000 complete [12823.115610] Lustre: server umount lustre-MDT0000 complete [12837.998264] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [12840.503989] Key type lgssc unregistered [12840.826397] LNet: 431809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12840.839715] LNetError: 431809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12840.870473] LNet: Removed LNI 192.168.204.149@tcp [12841.991189] Key type .llcrypt unregistered [12841.993151] Key type ._llcrypt unregistered [12872.147304] Key type ._llcrypt registered [12872.159887] Key type .llcrypt registered [12872.399522] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [12889.908992] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [12890.960696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12891.000159] alg: No test for adler32 (adler32-zlib) [12892.081179] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [12892.418208] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [12894.128368] Key type lgssc registered [12895.547124] Lustre: Echo OBD driver; http://www.lustre.org/ [12936.061778] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [12952.253791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12952.295611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12953.519318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12953.549420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12953.653776] Lustre: lustre-MDT0000: new disk, initializing [12953.756306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12953.773711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12957.930405] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12971.564413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12971.775307] Lustre: lustre-OST0000: new disk, initializing [12971.778403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12971.781932] Lustre: Skipped 1 previous similar message [12971.785254] Lustre: 436841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12971.853142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12978.340437] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [12981.364639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12981.373732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12981.405756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12987.437938] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13003.509694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13003.671570] Lustre: 435823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13003.776375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13003.891581] Lustre: lustre-MDT0001: new disk, initializing [13004.085211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13004.115041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13004.131791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13009.366435] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [13017.999471] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13018.179860] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13022.512466] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13022.682819] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13028.841597] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13028.854933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13028.870695] Lustre: Skipped 1 previous similar message [13031.049519] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [13033.715955] Lustre: server umount lustre-MDT0001 complete [13039.093943] LustreError: 435831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13039.122506] LustreError: 435831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13044.206135] LustreError: 437779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13044.230137] LustreError: 437779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13049.314474] LustreError: 435831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13049.327417] LustreError: 435831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13052.902217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13052.908322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13052.925928] Lustre: Skipped 1 previous similar message [13052.930167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13054.435583] LustreError: 435836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13058.555937] Lustre: server umount lustre-OST0000 complete [13059.559673] LustreError: 435835:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13068.677713] Lustre: server umount lustre-MDT0000 complete [13082.097210] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [13084.999312] Key type lgssc unregistered [13085.328756] LNet: 440589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13085.334784] LNetError: 440589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13085.355917] LNet: Removed LNI 192.168.204.149@tcp [13086.435583] Key type .llcrypt unregistered [13086.437840] Key type ._llcrypt unregistered [13125.501961] Key type ._llcrypt registered [13125.504380] Key type .llcrypt registered [13125.639782] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [13125.988827] Key type .llcrypt unregistered [13125.992585] Key type ._llcrypt unregistered [13142.634825] Key type ._llcrypt registered [13142.636632] Key type .llcrypt registered [13142.728669] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [13143.636588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13143.688832] alg: No test for adler32 (adler32-zlib) [13144.721723] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [13144.939533] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [13146.578466] Key type lgssc registered [13147.576254] Lustre: Echo OBD driver; http://www.lustre.org/ [13176.327171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13176.373247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13177.664727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13177.702918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13177.796612] Lustre: lustre-MDT0000: new disk, initializing [13177.887729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13177.919437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13182.115385] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [13191.939793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13192.070130] Lustre: 443958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13192.125732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13192.129953] Lustre: Skipped 1 previous similar message [13192.199700] Lustre: lustre-MDT0001: new disk, initializing [13192.318481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13192.358422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13192.366772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13196.854277] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [13205.921457] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13211.194885] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13217.556903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13217.989727] Lustre: lustre-OST0000: new disk, initializing [13217.999263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13218.005835] Lustre: 445924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13218.063142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13223.209064] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [13223.991915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13224.015663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13224.140074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13233.567427] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13243.378303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13243.387397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13243.421241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13244.898987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13244.899233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13244.926647] Lustre: Skipped 1 previous similar message [13244.932490] Lustre: Skipped 1 previous similar message [13247.482976] Lustre: server umount lustre-MDT0000 complete [13251.408448] LustreError: 443951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948853 with bad export cookie 13021189690511689422 [13251.433951] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13251.841947] Lustre: server umount lustre-MDT0001 complete [13266.085708] Lustre: server umount lustre-OST0000 complete [13277.289991] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:14:38 (1788948878) [13285.609623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13286.145918] LustreError: 447953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13286.166281] LustreError: 447953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13286.208362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13291.406450] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [13291.501063] LustreError: 447954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13296.622438] LustreError: 447953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13300.117919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13300.471058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13304.590440] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [13311.946836] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13317.724135] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13336.871715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13337.068706] Lustre: Found index 0 for lustre-OST0000, updating log [13337.244317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13343.158561] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [13349.310424] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13357.749260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13367.276432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13367.281606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13367.293161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13367.296586] Lustre: Skipped 1 previous similar message [13372.040718] Lustre: server umount lustre-OST0000 complete [13388.597785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13388.599481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13388.621367] Lustre: Skipped 2 previous similar messages [13388.639464] Lustre: Skipped 4 previous similar messages [13393.890047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13394.919098] LustreError: 447977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13395.074605] Lustre: server umount lustre-MDT0000 complete [13398.938869] LustreError: 447935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949001 with bad export cookie 13021189690511690815 [13398.946788] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13398.951806] LustreError: 447935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13399.405442] Lustre: server umount lustre-MDT0001 complete [13408.100371] LNet: 1 local NIs in recovery (showing 1): 192.168.204.149@tcp [13408.306578] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:16:49 (1788949009) [14029.718540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14030.248181] LustreError: 454388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14030.268128] LustreError: 454388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [14030.345609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14035.370831] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [14035.433079] LustreError: 454389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14040.548903] LustreError: 454388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14043.941607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14044.390752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14049.536896] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [14057.679582] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14063.712313] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14083.771035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14084.141434] Lustre: Found index 0 for lustre-OST0000, updating log [14084.489832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14085.984332] LNet: 1 local NIs in recovery (showing 1): 192.168.204.149@tcp [14090.522680] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [14098.325043] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14103.017409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14103.022029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14103.043912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14103.055400] Lustre: Skipped 1 previous similar message [14106.796962] Lustre: server umount lustre-OST0000 complete [14111.207222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14111.230690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14111.248905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14112.302878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14113.383604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14113.397323] Lustre: Skipped 3 previous similar messages [14116.660694] Lustre: server umount lustre-MDT0000 complete [14117.478070] LustreError: 455117:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14117.504957] LustreError: 455117:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14120.145603] LustreError: 454371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949722 with bad export cookie 13021189690511691900 [14120.157923] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14120.158750] LustreError: 454371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14120.587739] Lustre: server umount lustre-MDT0001 complete [14146.407274] LNet: 1 local NIs in recovery (showing 1): 192.168.204.149@tcp [14150.711871] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [14160.796776] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [14201.129425] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:30:02 (1788949802) [14220.606203] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [14223.531345] Key type lgssc unregistered [14223.881845] LNet: 461006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14223.894438] LNetError: 461006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14345.774596] LNet: Removed LNI 192.168.204.149@tcp [14347.055602] Key type .llcrypt unregistered [14347.061181] Key type ._llcrypt unregistered [14369.557383] Key type ._llcrypt registered [14369.559877] Key type .llcrypt registered [14369.699793] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [14370.768893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14370.859816] alg: No test for adler32 (adler32-zlib) [14372.146651] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [14372.393409] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [14374.040239] Key type lgssc registered [14375.454298] Lustre: Echo OBD driver; http://www.lustre.org/ [14380.894483] LNet: Added LNI 192.168.204.149@tcp100 [8/256/0/180] [14381.865810] LNet: Added LNI 192.168.204.149@tcp101 [8/256/0/180] [14382.909070] LNet: Added LNI 192.168.204.149@tcp102 [8/256/0/180] [14384.107481] LNet: Added LNI 192.168.204.149@tcp103 [8/256/0/180] [14385.074817] LNet: Added LNI 192.168.204.149@tcp104 [8/256/0/180] [14386.004503] LNet: Added LNI 192.168.204.149@tcp105 [8/256/0/180] [14386.902461] LNet: Added LNI 192.168.204.149@tcp106 [8/256/0/180] [14388.027521] LNet: Added LNI 192.168.204.149@tcp107 [8/256/0/180] [14389.424718] LNet: Added LNI 192.168.204.149@tcp108 [8/256/0/180] [14390.577334] LNet: Added LNI 192.168.204.149@tcp109 [8/256/0/180] [14391.458360] LNet: Added LNI 192.168.204.149@tcp110 [8/256/0/180] [14392.507838] LNet: Added LNI 192.168.204.149@tcp111 [8/256/0/180] [14393.524675] LNet: Added LNI 192.168.204.149@tcp112 [8/256/0/180] [14394.500914] LNet: Added LNI 192.168.204.149@tcp113 [8/256/0/180] [14395.579950] LNet: Added LNI 192.168.204.149@tcp114 [8/256/0/180] [14396.720460] LNet: Added LNI 192.168.204.149@tcp115 [8/256/0/180] [14397.814414] LNet: Added LNI 192.168.204.149@tcp116 [8/256/0/180] [14398.645411] LNet: Added LNI 192.168.204.149@tcp117 [8/256/0/180] [14399.558660] LNet: Added LNI 192.168.204.149@tcp118 [8/256/0/180] [14400.732612] LNet: Added LNI 192.168.204.149@tcp119 [8/256/0/180] [14401.657137] LNet: Added LNI 192.168.204.149@tcp120 [8/256/0/180] [14402.638670] LNet: Added LNI 192.168.204.149@tcp121 [8/256/0/180] [14403.718028] LNet: Added LNI 192.168.204.149@tcp122 [8/256/0/180] [14404.616632] LNet: Added LNI 192.168.204.149@tcp123 [8/256/0/180] [14405.508300] LNet: Added LNI 192.168.204.149@tcp124 [8/256/0/180] [14406.310252] LNet: Added LNI 192.168.204.149@tcp125 [8/256/0/180] [14407.247823] LNet: Added LNI 192.168.204.149@tcp126 [8/256/0/180] [14408.110493] LNet: Added LNI 192.168.204.149@tcp127 [8/256/0/180] [14409.030793] LNet: Added LNI 192.168.204.149@tcp128 [8/256/0/180] [14410.061678] LNet: Added LNI 192.168.204.149@tcp129 [8/256/0/180] [14411.268946] LNet: Added LNI 192.168.204.149@tcp130 [8/256/0/180] [14412.311308] LNet: Added LNI 192.168.204.149@tcp131 [8/256/0/180] [14413.086226] LNet: Added LNI 192.168.204.149@tcp132 [8/256/0/180] [14413.868550] LNet: Added LNI 192.168.204.149@tcp133 [8/256/0/180] [14414.594974] LNet: Added LNI 192.168.204.149@tcp134 [8/256/0/180] [14415.553716] LNet: Added LNI 192.168.204.149@tcp135 [8/256/0/180] [14422.689056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14422.716155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14423.886667] Lustre: 464035:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.149@tcp: can't add NID 192.168.204.149@tcp130: rc = -75 [14423.982584] LNetError: 464059:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.149 [14423.995047] LNetError: Protocol error connecting to 192.168.204.149@tcp119 at host 192.168.204.149:988: is it running a compatible version of Lustre? [14424.009655] LNetError: Skipped 10 previous similar messages [14424.062749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14424.087723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14424.211809] Lustre: lustre-MDT0000: new disk, initializing [14424.342757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14424.373181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14424.609292] LNet: 8 local NIs in recovery (showing 8): 192.168.204.149@tcp122, 192.168.204.149@tcp123, 192.168.204.149@tcp126, 192.168.204.149@tcp128, 192.168.204.149@tcp129, 192.168.204.149@tcp127, 192.168.204.149@tcp124, 192.168.204.149@tcp119 [14424.612197] LNetError: 464071:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.149 [14424.641815] LNetError: 464071:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [14424.654749] LNetError: Protocol error connecting to 192.168.204.149@tcp122 at host 192.168.204.149:988: is it running a compatible version of Lustre? [14425.698098] LNetError: 464068:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.149 [14425.737533] LNetError: Protocol error connecting to 192.168.204.149@tcp124 at host 192.168.204.149:988: is it running a compatible version of Lustre? [14425.768654] LNetError: Skipped 1 previous similar message [14427.745545] LNetError: 464058:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.149 [14427.755398] LNetError: 464058:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14427.765195] LNetError: Protocol error connecting to 192.168.204.149@tcp128 at host 192.168.204.149:988: is it running a compatible version of Lustre? [14427.784388] LNetError: Skipped 1 previous similar message [14430.168258] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [14431.842615] LNetError: 464068:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.149 [14431.855111] LNetError: 464068:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14431.871670] LNetError: Protocol error connecting to 192.168.204.149@tcp128 at host 192.168.204.149:988: is it running a compatible version of Lustre? [14431.888325] LNetError: Skipped 1 previous similar message [14445.680277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14445.876379] Lustre: 464050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14445.964499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14445.970687] Lustre: Skipped 1 previous similar message [14446.027445] Lustre: lustre-MDT0001: new disk, initializing [14446.060598] Lustre: 464856:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.149@tcp: can't add NID 192.168.204.149@tcp130: rc = -75 [14446.078973] Lustre: 464856:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14446.224079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14446.284452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14446.305312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14450.911299] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [14454.819504] Lustre: 465266:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.149@tcp: can't add NID 192.168.204.149@tcp130: rc = -75 [14454.838440] Lustre: 465266:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14455.336220] LNetError: 464061:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.149 [14455.359354] LNetError: 464061:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14455.365956] LNetError: Protocol error connecting to 192.168.204.149@tcp124 at host 192.168.204.149:988: is it running a compatible version of Lustre? [14455.376039] LNetError: Skipped 2 previous similar messages [14461.236366] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14467.479263] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14472.802084] LNetError: 461492:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.149 [14472.815322] LNetError: 461492:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14472.825440] LNetError: Protocol error connecting to 192.168.204.149@tcp117 at host 192.168.204.149:988: is it running a compatible version of Lustre? [14472.836717] LNetError: Skipped 9 previous similar messages [14473.889195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14474.362773] Lustre: lustre-OST0000: new disk, initializing [14474.380313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14474.390429] Lustre: 466027:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14474.559737] Lustre: 466032:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.149@tcp: can't add NID 192.168.204.149@tcp130: rc = -75 [14474.579796] Lustre: 466032:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14474.672599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14482.512205] Lustre: 466338:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.149@tcp: can't add NID 192.168.204.149@tcp130: rc = -75 [14482.527534] Lustre: 466338:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14482.638682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14482.654771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14482.935729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14483.346641] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [14484.065983] LNet: 2 local NIs in recovery (showing 2): 192.168.204.149@tcp128, 192.168.204.149@tcp122 [14493.736211] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14503.395861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14503.398527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14503.405107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14503.419217] Lustre: Skipped 1 previous similar message [14503.430327] Lustre: Skipped 1 previous similar message [14508.516112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14508.521649] Lustre: Skipped 1 previous similar message [14509.374676] Lustre: server umount lustre-OST0000 complete [14510.692064] LNetError: 461494:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.149 [14510.705644] LNetError: 461494:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [14510.728211] LNetError: Protocol error connecting to 192.168.204.149@tcp117 at host 192.168.204.149:988: is it running a compatible version of Lustre? [14510.749687] LNetError: Skipped 16 previous similar messages [14517.730711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14517.750404] LustreError: Skipped 1 previous similar message [14517.757029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14517.787633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14522.050791] Lustre: server umount lustre-MDT0000 complete [14525.891658] LustreError: 464045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788950127 with bad export cookie 11355561849858679096 [14525.893874] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14525.916749] LustreError: 464045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14526.356027] Lustre: server umount lustre-MDT0001 complete [14538.704241] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [14542.403051] Key type lgssc unregistered [14542.913858] LNet: 467942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14542.924660] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14542.933340] LNet: Removed LNI 192.168.204.149@tcp [14543.968234] LNet: Removed LNI 192.168.204.149@tcp100 [14543.974990] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14543.992425] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14544.015476] LNet: Removed LNI 192.168.204.149@tcp101 [14545.057102] LNet: Removed LNI 192.168.204.149@tcp102 [14545.066072] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14545.075126] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14545.104412] LNet: Removed LNI 192.168.204.149@tcp103 [14546.144415] LNet: Removed LNI 192.168.204.149@tcp104 [14547.168205] LNet: Removed LNI 192.168.204.149@tcp105 [14547.174397] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14547.222867] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14547.247964] LNet: Removed LNI 192.168.204.149@tcp106 [14548.258316] LNet: Removed LNI 192.168.204.149@tcp107 [14549.345839] LNet: Removed LNI 192.168.204.149@tcp108 [14550.368413] LNet: Removed LNI 192.168.204.149@tcp109 [14551.392148] LNet: Removed LNI 192.168.204.149@tcp110 [14551.397564] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14551.402792] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14551.407573] LNet: Removed LNI 192.168.204.149@tcp111 [14552.418126] LNet: Removed LNI 192.168.204.149@tcp112 [14553.442523] LNet: Removed LNI 192.168.204.149@tcp113 [14554.465696] LNet: Removed LNI 192.168.204.149@tcp114 [14555.490553] LNet: Removed LNI 192.168.204.149@tcp115 [14556.513783] LNet: Removed LNI 192.168.204.149@tcp116 [14557.536231] LNet: Removed LNI 192.168.204.149@tcp117 [14558.560987] LNet: Removed LNI 192.168.204.149@tcp118 [14559.584166] LNet: Removed LNI 192.168.204.149@tcp119 [14559.594582] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14559.619791] LNetError: 467942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14559.632022] LNet: Removed LNI 192.168.204.149@tcp120 [14560.672389] LNet: Removed LNI 192.168.204.149@tcp121 [14560.674172] LNet: Removed LNI 192.168.204.149@tcp122 [14561.696304] LNet: Removed LNI 192.168.204.149@tcp123 [14562.720141] LNet: Removed LNI 192.168.204.149@tcp124 [14563.744178] LNet: Removed LNI 192.168.204.149@tcp125 [14564.768200] LNet: Removed LNI 192.168.204.149@tcp126 [14565.792450] LNet: Removed LNI 192.168.204.149@tcp127 [14565.799870] LNet: Removed LNI 192.168.204.149@tcp128 [14566.883554] LNet: Removed LNI 192.168.204.149@tcp129 [14566.888064] LNet: Removed LNI 192.168.204.149@tcp130 [14566.898041] LNet: Removed LNI 192.168.204.149@tcp131 [14566.904240] LNet: Removed LNI 192.168.204.149@tcp132 [14566.908820] LNet: Removed LNI 192.168.204.149@tcp133 [14566.912106] LNet: Removed LNI 192.168.204.149@tcp134 [14566.925078] LNet: Removed LNI 192.168.204.149@tcp135 [14567.910086] Key type .llcrypt unregistered [14567.912064] Key type ._llcrypt unregistered [14593.631811] Key type ._llcrypt registered [14593.633538] Key type .llcrypt registered [14593.723727] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [14610.452777] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [14612.086624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14612.211734] alg: No test for adler32 (adler32-zlib) [14613.627371] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [14613.857803] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [14615.497452] Key type lgssc registered [14616.674848] Lustre: Echo OBD driver; http://www.lustre.org/ [14660.559489] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:37:40 (1788950260) [14684.596624] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [14687.456495] Key type lgssc unregistered [14687.981250] LNet: 472029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14687.991522] LNetError: 472029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14688.007962] LNet: Removed LNI 192.168.204.149@tcp [14689.277734] Key type .llcrypt unregistered [14689.282073] Key type ._llcrypt unregistered [14707.896968] Key type ._llcrypt registered [14707.903319] Key type .llcrypt registered [14708.063414] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [14709.270433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14709.507958] alg: No test for adler32 (adler32-zlib) [14710.634416] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [14711.010932] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [14712.792213] Key type lgssc registered [14714.215722] Lustre: Echo OBD driver; http://www.lustre.org/ [14723.716912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14723.771232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14725.075327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14725.120841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14725.192474] Lustre: lustre-MDT0000: new disk, initializing [14725.269806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14725.288282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14731.159320] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [14741.137733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14741.274735] Lustre: 473237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14741.324418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14741.328890] Lustre: Skipped 1 previous similar message [14741.389662] Lustre: lustre-MDT0001: new disk, initializing [14741.485321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14741.517246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14741.532887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14745.940678] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [14755.277612] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14760.801633] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14767.523580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14767.881630] Lustre: lustre-OST0000: new disk, initializing [14767.884568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14767.899597] Lustre: 475209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14768.016977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14773.273417] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [14777.935459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14777.955632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14778.111993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14784.083320] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14788.608458] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing lnet_if_list [14795.484956] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14795.550757] LNet: Added LNI 192.168.204.149@tcp10 [8/256/0/180] [14802.744827] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14802.783580] LNet: Added LNI 192.168.204.149@tcp11 [8/256/0/180] [14807.975843] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14807.998561] LNet: Added LNI 192.168.204.149@tcp12 [8/256/0/180] [14811.762171] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14811.803029] LNet: Added LNI 192.168.204.149@tcp13 [8/256/0/180] [14816.364425] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14816.402656] LNet: Added LNI 192.168.204.149@tcp14 [8/256/0/180] [14821.068391] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14821.100666] LNet: Added LNI 192.168.204.149@tcp15 [8/256/0/180] [14826.573305] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14826.640541] LNet: Added LNI 192.168.204.149@tcp16 [8/256/0/180] [14835.079240] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14835.114803] LNet: Added LNI 192.168.204.149@tcp17 [8/256/0/180] [14843.000864] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14843.078994] LNet: Added LNI 192.168.204.149@tcp18 [8/256/0/180] [14848.518749] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14848.542814] LNet: Added LNI 192.168.204.149@tcp19 [8/256/0/180] [14853.238880] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14853.307099] LNet: Added LNI 192.168.204.149@tcp20 [8/256/0/180] [14859.056909] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14859.097722] LNet: Added LNI 192.168.204.149@tcp21 [8/256/0/180] [14865.552793] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14865.591650] LNet: Added LNI 192.168.204.149@tcp22 [8/256/0/180] [14871.416963] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14871.451500] LNet: Added LNI 192.168.204.149@tcp23 [8/256/0/180] [14876.939639] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14876.988151] LNet: Added LNI 192.168.204.149@tcp24 [8/256/0/180] [14883.461719] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14883.515921] LNet: Added LNI 192.168.204.149@tcp25 [8/256/0/180] [14889.683442] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14889.712220] LNet: Added LNI 192.168.204.149@tcp26 [8/256/0/180] [14895.752433] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14895.799779] LNet: Added LNI 192.168.204.149@tcp27 [8/256/0/180] [14902.834607] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14902.872760] LNet: Added LNI 192.168.204.149@tcp28 [8/256/0/180] [14909.712740] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14909.736910] LNet: Added LNI 192.168.204.149@tcp29 [8/256/0/180] [14915.786126] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14915.807267] LNet: Added LNI 192.168.204.149@tcp30 [8/256/0/180] [14922.100969] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14922.136707] LNet: Added LNI 192.168.204.149@tcp31 [8/256/0/180] [14928.157871] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14928.212345] LNet: Added LNI 192.168.204.149@tcp32 [8/256/0/180] [14935.745368] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14935.778913] LNet: Added LNI 192.168.204.149@tcp33 [8/256/0/180] [14943.585991] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14943.627562] LNet: Added LNI 192.168.204.149@tcp34 [8/256/0/180] [14951.211597] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14951.247720] LNet: Added LNI 192.168.204.149@tcp35 [8/256/0/180] [14958.883262] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14958.913275] LNet: Added LNI 192.168.204.149@tcp36 [8/256/0/180] [14965.755345] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14965.793198] LNet: Added LNI 192.168.204.149@tcp37 [8/256/0/180] [14973.428981] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14973.469683] LNet: Added LNI 192.168.204.149@tcp38 [8/256/0/180] [14980.264258] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14980.303903] LNet: Added LNI 192.168.204.149@tcp39 [8/256/0/180] [14988.870985] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14988.908783] LNet: Added LNI 192.168.204.149@tcp40 [8/256/0/180] [14997.746701] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14997.775254] LNet: Added LNI 192.168.204.149@tcp41 [8/256/0/180] [15005.885186] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15005.912531] LNet: Added LNI 192.168.204.149@tcp42 [8/256/0/180] [15014.466588] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15014.504752] LNet: Added LNI 192.168.204.149@tcp43 [8/256/0/180] [15023.513815] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15023.555965] LNet: Added LNI 192.168.204.149@tcp44 [8/256/0/180] [15032.249487] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15032.284210] LNet: Added LNI 192.168.204.149@tcp45 [8/256/0/180] [15040.602868] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15040.676432] LNet: Added LNI 192.168.204.149@tcp46 [8/256/0/180] [15049.456534] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15049.516827] LNet: Added LNI 192.168.204.149@tcp47 [8/256/0/180] [15060.122695] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15060.219416] LNet: Added LNI 192.168.204.149@tcp48 [8/256/0/180] [15069.811251] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15069.863537] LNet: Added LNI 192.168.204.149@tcp49 [8/256/0/180] [15078.751714] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15078.788805] LNet: Added LNI 192.168.204.149@tcp50 [8/256/0/180] [15099.874328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15099.884344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15099.902288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15100.897970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15100.899415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15102.336070] Lustre: server umount lustre-MDT0000 complete [15106.129384] LustreError: 473232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788950708 with bad export cookie 6051105555676937645 [15106.137530] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15106.149271] LustreError: 473232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15106.559659] Lustre: server umount lustre-MDT0001 complete [15125.016697] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [15127.520148] Lustre: 472511:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788950713/real 1788950713] req@000000001774cbf4 x1875850363580544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788950729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15127.554518] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15127.590332] Lustre: Skipped 1 previous similar message [15132.705734] Lustre: 472511:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788950718/real 1788950718] req@00000000f41c6811 x1875850363580672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788950734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15134.616347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15137.760770] Lustre: 472510:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788950723/real 1788950723] req@000000002bc5b4d3 x1875850363580928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788950739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15142.880117] Lustre: 472509:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788950728/real 1788950728] req@00000000e57814d5 x1875850363581056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788950744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15151.363017] LustreError: 475203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15151.371827] LustreError: 475203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15151.417763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15152.418753] LustreError: 499454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15152.436907] LustreError: 499454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15154.765979] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15156.515081] LustreError: 499452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15156.517314] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15156.536578] LustreError: 499452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15161.826317] LustreError: 475204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15161.841140] LustreError: 475204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15162.019035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15162.411753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15167.496753] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15167.680726] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15177.104615] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15183.551587] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15207.907598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15207.915206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15207.929420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15207.945148] Lustre: Skipped 2 previous similar messages [15209.967854] Lustre: server umount lustre-OST0000 complete [15218.659905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15218.666433] LustreError: Skipped 1 previous similar message [15218.669477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15218.673240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15218.681269] Lustre: Skipped 2 previous similar messages [15218.684744] Lustre: Skipped 2 previous similar messages [15223.777578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15223.796223] Lustre: Skipped 2 previous similar messages [15228.384283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15228.546691] Lustre: server umount lustre-MDT0000 complete [15228.904412] LustreError: 499452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15228.933383] LustreError: 499452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15232.932586] LustreError: 473232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788950835 with bad export cookie 6051105555676940298 [15232.946303] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15232.956548] LustreError: 473232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15233.837234] Lustre: server umount lustre-MDT0001 complete [15247.770884] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [15251.508235] Key type lgssc unregistered [15252.036382] LNet: 502252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15252.074161] LNetError: 502252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15253.089066] LNet: Removed LNI 192.168.204.149@tcp [15253.094630] LNetError: 502252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15253.106534] LNet: Removed LNI 192.168.204.149@tcp10 [15253.111624] LNet: Removed LNI 192.168.204.149@tcp11 [15253.116238] LNet: Removed LNI 192.168.204.149@tcp12 [15253.119771] LNet: Removed LNI 192.168.204.149@tcp13 [15253.122885] LNet: Removed LNI 192.168.204.149@tcp14 [15253.126488] LNet: Removed LNI 192.168.204.149@tcp15 [15253.130110] LNet: Removed LNI 192.168.204.149@tcp16 [15253.133500] LNet: Removed LNI 192.168.204.149@tcp17 [15253.136560] LNet: Removed LNI 192.168.204.149@tcp18 [15253.140227] LNet: Removed LNI 192.168.204.149@tcp19 [15253.142713] LNet: Removed LNI 192.168.204.149@tcp20 [15253.145416] LNet: Removed LNI 192.168.204.149@tcp21 [15253.148206] LNet: Removed LNI 192.168.204.149@tcp22 [15253.150843] LNet: Removed LNI 192.168.204.149@tcp23 [15253.154908] LNet: Removed LNI 192.168.204.149@tcp24 [15253.157988] LNet: Removed LNI 192.168.204.149@tcp25 [15253.159282] LNet: Removed LNI 192.168.204.149@tcp26 [15253.164654] LNet: Removed LNI 192.168.204.149@tcp27 [15253.167525] LNet: Removed LNI 192.168.204.149@tcp28 [15253.169904] LNet: Removed LNI 192.168.204.149@tcp29 [15253.173617] LNet: Removed LNI 192.168.204.149@tcp30 [15253.176930] LNet: Removed LNI 192.168.204.149@tcp31 [15253.180411] LNet: Removed LNI 192.168.204.149@tcp32 [15253.182922] LNet: Removed LNI 192.168.204.149@tcp33 [15253.185600] LNet: Removed LNI 192.168.204.149@tcp34 [15253.191524] LNet: Removed LNI 192.168.204.149@tcp35 [15253.194410] LNet: Removed LNI 192.168.204.149@tcp36 [15253.196944] LNet: Removed LNI 192.168.204.149@tcp37 [15253.199753] LNet: Removed LNI 192.168.204.149@tcp38 [15253.202636] LNet: Removed LNI 192.168.204.149@tcp39 [15253.205960] LNet: Removed LNI 192.168.204.149@tcp40 [15253.208802] LNet: Removed LNI 192.168.204.149@tcp41 [15253.211473] LNet: Removed LNI 192.168.204.149@tcp42 [15253.214693] LNet: Removed LNI 192.168.204.149@tcp43 [15253.217622] LNet: Removed LNI 192.168.204.149@tcp44 [15253.220434] LNet: Removed LNI 192.168.204.149@tcp45 [15253.224986] LNet: Removed LNI 192.168.204.149@tcp46 [15253.243903] LNet: Removed LNI 192.168.204.149@tcp47 [15253.249030] LNet: Removed LNI 192.168.204.149@tcp48 [15253.262892] LNet: Removed LNI 192.168.204.149@tcp49 [15253.286736] LNet: Removed LNI 192.168.204.149@tcp50 [15254.332610] Key type .llcrypt unregistered [15254.334983] Key type ._llcrypt unregistered [15273.908654] Key type ._llcrypt registered [15273.910768] Key type .llcrypt registered [15273.988484] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [15288.517766] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [15289.294284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15289.386042] alg: No test for adler32 (adler32-zlib) [15290.362668] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [15290.575321] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [15292.248364] Key type lgssc registered [15293.263872] Lustre: Echo OBD driver; http://www.lustre.org/ [15326.766753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15326.795032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15328.026259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15328.065597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15328.185667] Lustre: lustre-MDT0000: new disk, initializing [15328.257080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15328.275704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15332.911510] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15342.971617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15343.094848] Lustre: 505446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15343.141730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15343.146181] Lustre: Skipped 1 previous similar message [15343.206630] Lustre: lustre-MDT0001: new disk, initializing [15343.329809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15343.362048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15343.378655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15347.717491] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15356.849734] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15362.673825] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15371.182457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15371.390160] Lustre: *** cfs_fail_loc=90f, val=0*** [15371.514831] Lustre: lustre-OST0000: new disk, initializing [15371.521714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15371.532820] Lustre: 507463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15371.654187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15376.987253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15376.996833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15377.136247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15378.723343] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15388.778654] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15397.871262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15397.873060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15397.877186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15397.885465] Lustre: Skipped 1 previous similar message [15397.901699] LustreError: Skipped 1 previous similar message [15402.333137] Lustre: server umount lustre-OST0000 complete [15408.099136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15408.108712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15408.110375] Lustre: Skipped 1 previous similar message [15408.123983] Lustre: Skipped 2 previous similar messages [15412.791107] Lustre: server umount lustre-MDT0000 complete [15416.370505] LustreError: 505438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951018 with bad export cookie 6593656647184012010 [15416.382568] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15416.387292] LustreError: 505438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15416.801217] Lustre: server umount lustre-MDT0001 complete [15427.068424] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [15429.846746] Key type lgssc unregistered [15430.096649] LNet: 509367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15430.105357] LNetError: 509367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15430.123089] LNet: Removed LNI 192.168.204.149@tcp [15431.122540] Key type .llcrypt unregistered [15431.124234] Key type ._llcrypt unregistered [15451.538056] Key type ._llcrypt registered [15451.547786] Key type .llcrypt registered [15451.696097] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [15452.698894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15452.726759] alg: No test for adler32 (adler32-zlib) [15453.888516] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [15454.179911] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [15455.824231] Key type lgssc registered [15456.868447] Lustre: Echo OBD driver; http://www.lustre.org/ [15463.992499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15464.003259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15465.448932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15469.755667] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15475.688641] LustreError: 510633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15475.714628] LustreError: 510633:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15478.591897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15479.105106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15484.345164] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15491.845018] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15497.509947] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15504.661839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15505.127811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15512.207816] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15520.252546] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15534.605495] loop0: detected capacity change from 0 to 204800000 [15534.844989] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15534.880210] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15534.893778] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15541.009345] loop0: detected capacity change from 0 to 204800000 [15541.250885] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15541.274361] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15541.286055] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15546.467822] loop0: detected capacity change from 0 to 204800000 [15551.903318] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15552.045724] Lustre: fs15246-OST0000: new disk, initializing [15552.049392] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15552.060949] Lustre: 514227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15552.121222] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15559.006812] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15569.694169] loop1: detected capacity change from 0 to 204800000 [15575.155177] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15575.267488] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15575.320956] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15575.392446] Lustre: fs15246-MDT0000: new disk, initializing [15575.602953] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15575.640340] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15575.829810] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15576.865675] Lustre: 515349:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000fb3be6f0 x1875851142544256/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788951194 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15580.213346] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15585.957000] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15585.971868] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15585.993853] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15604.210071] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15604.224966] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15606.721996] Lustre: server umount fs15246-MDT0000 complete [15609.320859] LustreError: 511875:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15614.458910] LustreError: 510629:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15615.071982] Lustre: server umount fs15246-OST0000 complete [15629.806930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15629.829605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15629.836682] Lustre: Skipped 1 previous similar message [15634.863205] Lustre: server umount lustre-OST0000 complete [15640.034321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15640.045395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15640.056147] Lustre: Skipped 2 previous similar messages [15644.952684] Lustre: server umount lustre-MDT0000 complete [15645.157657] LustreError: 510634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15645.191492] LustreError: 510634:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15648.746547] LustreError: 510614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951250 with bad export cookie 6205613840730685723 [15648.771895] LustreError: 510614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15648.772558] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15649.131397] Lustre: server umount lustre-MDT0001 complete [15659.765808] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [15662.935860] Key type lgssc unregistered [15663.237870] LNet: 518449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15663.248766] LNetError: 518449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15663.274920] LNet: Removed LNI 192.168.204.149@tcp [15664.084258] Key type .llcrypt unregistered [15664.090759] Key type ._llcrypt unregistered [15684.903481] Key type ._llcrypt registered [15684.904746] Key type .llcrypt registered [15685.024946] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [15686.138389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15686.170438] alg: No test for adler32 (adler32-zlib) [15687.336754] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [15687.694750] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [15689.416200] Key type lgssc registered [15690.595584] Lustre: Echo OBD driver; http://www.lustre.org/ [15727.239418] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [15735.845053] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [15766.673971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15766.715453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15767.873860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15767.901797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15767.976299] Lustre: lustre-MDT0000: new disk, initializing [15768.069709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15768.089640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15772.156694] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15781.823363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15781.923342] Lustre: 522908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15781.997634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15782.000126] Lustre: Skipped 1 previous similar message [15782.065402] Lustre: lustre-MDT0001: new disk, initializing [15782.142477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15782.191252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15782.203762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15786.812722] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15796.077702] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15801.427790] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15807.982154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15808.382601] Lustre: lustre-OST0000: new disk, initializing [15808.391029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15808.400886] Lustre: 524878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15808.522862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15815.784896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15815.804519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15815.849858] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [15815.916604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15825.264320] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15829.847816] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15830.011780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15833.931801] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15834.135585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15836.646324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15836.646567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15836.648147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15836.658975] LustreError: Skipped 1 previous similar message [15836.683714] Lustre: Skipped 1 previous similar message [15841.840427] Lustre: server umount lustre-OST0000 complete [15846.886756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15846.891551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15846.907688] Lustre: Skipped 1 previous similar message [15846.910500] Lustre: Skipped 1 previous similar message [15851.901994] Lustre: server umount lustre-MDT0000 complete [15852.002299] LustreError: 524364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15852.013410] LustreError: 524364:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15854.896847] LustreError: 522902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951456 with bad export cookie 98127883074517358 [15854.898064] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15854.907335] LustreError: 522902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15855.176843] Lustre: server umount lustre-MDT0001 complete [15878.377788] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [15888.581305] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [15918.229232] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15593 sec ========= 06:58:39 (1788951519) [15920.236967] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:58:41 (1788951521) === [15924.086959] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:58:44 (1788951524) === [15964.186779] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [15967.007484] Key type lgssc unregistered [15967.371758] LNet: 531520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15967.389879] LNetError: 531520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15968.423643] LNet: Removed LNI 192.168.204.149@tcp [15969.393145] Key type .llcrypt unregistered [15969.400401] Key type ._llcrypt unregistered