[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 458235527 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003263] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008401] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009011] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.012140] LSM: Security Framework initializing [ 0.013053] Yama: becoming mindful. [ 0.014031] SELinux: Initializing. [ 0.015048] *** VALIDATE selinux *** [ 0.023435] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027624] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] *** VALIDATE tmpfs *** [ 0.031498] *** VALIDATE proc *** [ 0.034220] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037076] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040034] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.044417] debug: unmapping init [mem 0xffffffff8fe59000-0xffffffff8fe60fff] [ 0.047169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048740] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053013] ... max period: 00007fffffffffff [ 0.054013] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056361] rcu: Hierarchical SRCU implementation. [ 0.058510] smp: Bringing up secondary CPUs ... [ 0.059651] x86: Booting SMP configuration: [ 0.060026] .... node #0, CPUs: #1 #2 #3 [ 0.073076] smp: Brought up 1 node, 4 CPUs [ 0.075014] smpboot: Max logical packages: 1 [ 0.076018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151136] node 0 deferred pages initialised in 73ms [ 0.154204] devtmpfs: initialized [ 0.155217] x86/mm: Memory block size: 128MB [ 0.157847] gcov: version magic: 0x41383552 [ 0.159499] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162158] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164450] pinctrl core: initialized pinctrl subsystem [ 0.167238] [ 0.167767] ************************************************************* [ 0.170022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173024] ** ** [ 0.175018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177023] ** ** [ 0.180021] ** This means that this kernel is built to expose internal ** [ 0.182017] ** IOMMU data structures, which may compromise security on ** [ 0.184022] ** your system. ** [ 0.187022] ** ** [ 0.189019] ** If you see this message and you are not debugging the ** [ 0.192021] ** kernel, report this immediately to your vendor! ** [ 0.194019] ** ** [ 0.197022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199019] ************************************************************* [ 0.201884] NET: Registered protocol family 16 [ 0.204511] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.209090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213088] cpuidle: using governor menu [ 0.214354] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216445] PCI: Using configuration type 1 for base access [ 0.218146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.229251] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.231028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235212] cryptd: max_cpu_qlen set to 1000 [ 0.238219] ACPI: Added _OSI(Module Device) [ 0.239024] ACPI: Added _OSI(Processor Device) [ 0.240026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246079] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252470] ACPI: Interpreter enabled [ 0.253056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255015] ACPI: Using IOAPIC for interrupt routing [ 0.257009] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261359] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271627] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.274042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284378] acpiphp: Slot [2] registered [ 0.286116] acpiphp: Slot [5] registered [ 0.287113] acpiphp: Slot [6] registered [ 0.288102] acpiphp: Slot [7] registered [ 0.290130] acpiphp: Slot [8] registered [ 0.291124] acpiphp: Slot [9] registered [ 0.292118] acpiphp: Slot [10] registered [ 0.294177] acpiphp: Slot [3] registered [ 0.295112] acpiphp: Slot [4] registered [ 0.296096] acpiphp: Slot [11] registered [ 0.297070] acpiphp: Slot [12] registered [ 0.298063] acpiphp: Slot [13] registered [ 0.299071] acpiphp: Slot [14] registered [ 0.300039] acpiphp: Slot [15] registered [ 0.302117] acpiphp: Slot [16] registered [ 0.303080] acpiphp: Slot [17] registered [ 0.304095] acpiphp: Slot [18] registered [ 0.305116] acpiphp: Slot [19] registered [ 0.307081] acpiphp: Slot [20] registered [ 0.308089] acpiphp: Slot [21] registered [ 0.309101] acpiphp: Slot [22] registered [ 0.310114] acpiphp: Slot [23] registered [ 0.312094] acpiphp: Slot [24] registered [ 0.313068] acpiphp: Slot [25] registered [ 0.314046] acpiphp: Slot [26] registered [ 0.315105] acpiphp: Slot [27] registered [ 0.316084] acpiphp: Slot [28] registered [ 0.317013] acpiphp: Slot [29] registered [ 0.317926] acpiphp: Slot [30] registered [ 0.319079] acpiphp: Slot [31] registered [ 0.320058] PCI host bridge to bus 0000:00 [ 0.322021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.323018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.332023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335941] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339183] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.360658] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.362857] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.366046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369846] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.376017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402760] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.459823] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.466021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.473016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.497832] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.509017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.524021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.535238] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.540018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.547015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.560020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.568323] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.578016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.582012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.595017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.603904] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.612018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.629020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.638000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.638212] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.639000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.642369] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.644134] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.649029] iommu: Default domain type: Passthrough [ 0.650406] SCSI subsystem initialized [ 0.651097] ACPI: bus type USB registered [ 0.652007] usbcore: registered new interface driver usbfs [ 0.654063] usbcore: registered new interface driver hub [ 0.655050] usbcore: registered new device driver usb [ 0.656000] pps_core: LinuxPPS API ver. 1 registered [ 0.656000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.659064] PTP clock support registered [ 0.661144] EDAC MC: Ver: 3.0.0 [ 0.663115] PCI: Using ACPI for IRQ routing [ 0.665278] NetLabel: Initializing [ 0.666014] NetLabel: domain hash size = 128 [ 0.667009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.669085] NetLabel: unlabeled traffic allowed by default [ 0.671188] vgaarb: loaded [ 0.672231] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.673008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.681480] clocksource: Switched to clocksource kvm-clock [ 0.786385] VFS: Disk quotas dquot_6.6.0 [ 0.787868] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.790547] *** VALIDATE ramfs *** [ 0.791830] *** VALIDATE hugetlbfs *** [ 0.793306] pnp: PnP ACPI init [ 0.795702] pnp: PnP ACPI: found 6 devices [ 0.812735] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.814866] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.816225] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.817489] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.818880] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.820229] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.821928] NET: Registered protocol family 2 [ 0.823400] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.826374] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.828444] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.832200] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.834544] TCP: Hash tables configured (established 65536 bind 65536) [ 0.836418] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.838362] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.840324] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.842258] NET: Registered protocol family 1 [ 0.844659] RPC: Registered named UNIX socket transport module. [ 0.845759] RPC: Registered udp transport module. [ 0.846686] RPC: Registered tcp transport module. [ 0.847566] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.849771] NET: Registered protocol family 44 [ 0.850641] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.852515] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.854279] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.855588] PCI: CLS 0 bytes, default 64 [ 0.856610] Unpacking initramfs... [ 2.178138] debug: unmapping init [mem 0xffffa0ef7cc54000-0xffffa0ef7ffbffff] [ 2.182270] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.184429] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.186956] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.687660] Initialise system trusted keyrings [ 2.689250] Key type blacklist registered [ 2.691112] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.698826] zbud: loaded [ 2.701585] *** VALIDATE nfs *** [ 2.702425] *** VALIDATE nfs4 *** [ 2.703457] pstore: using deflate compression [ 2.706193] Platform Keyring initialized [ 2.792579] NET: Registered protocol family 38 [ 2.794418] Key type asymmetric registered [ 2.795734] Asymmetric key parser 'x509' registered [ 2.797617] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.800488] io scheduler mq-deadline registered [ 2.802016] io scheduler kyber registered [ 2.803745] io scheduler bfq registered [ 2.805779] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.808752] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.811059] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.813167] ACPI: Power Button [PWRF] [ 2.816687] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.821594] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.830535] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.836271] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.851674] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.879553] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.905690] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.909317] Non-volatile memory driver v1.3 [ 2.910683] Linux agpgart interface v0.103 [ 2.940773] virtio_blk virtio1: [vda] 150008 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.943468] vda: detected capacity change from 0 to 76804096 [ 2.958609] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.961472] vdb: detected capacity change from 0 to 1073741824 [ 2.977404] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.979174] vdc: detected capacity change from 0 to 2621440000 [ 2.992607] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.994975] vdd: detected capacity change from 0 to 2621440000 [ 3.007281] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.009911] vde: detected capacity change from 0 to 4294967296 [ 3.024522] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.027154] vdf: detected capacity change from 0 to 4294967296 [ 3.035501] libphy: Fixed MDIO Bus: probed [ 3.040825] usbcore: registered new interface driver usbserial_generic [ 3.042604] usbserial: USB Serial support registered for generic [ 3.044167] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.047268] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.048638] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.050278] mousedev: PS/2 mouse device common for all mice [ 3.052651] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.056110] rtc_cmos 00:05: RTC can wake from S4 [ 3.057222] rtc_cmos 00:05: registered as rtc0 [ 3.061140] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.063630] intel_pstate: CPU model not supported [ 3.065592] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.071841] hid: raw HID events driver (C) Jiri Kosina [ 3.074091] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.074472] usbcore: registered new interface driver usbhid [ 3.078154] usbhid: USB HID core driver [ 3.079645] drop_monitor: Initializing network drop monitor service [ 3.081614] Initializing XFRM netlink socket [ 3.082920] NET: Registered protocol family 10 [ 3.085718] Segment Routing with IPv6 [ 3.087209] NET: Registered protocol family 17 [ 3.088972] mpls_gso: MPLS GSO support [ 3.093253] RAS: Correctable Errors collector initialized. [ 3.095470] AVX version of gcm_enc/dec engaged. [ 3.097023] AES CTR mode by8 optimization enabled [ 3.162723] sched_clock: Marking stable (3162704304, 0)->(4130277897, -967573593) [ 3.165459] registered taskstats version 1 [ 3.167166] Loading compiled-in X.509 certificates [ 3.168688] zswap: loaded using pool lzo/zbud [ 3.189858] Key type big_key registered [ 3.200616] Key type encrypted registered [ 3.202146] ima: No TPM chip found, activating TPM-bypass! [ 3.203978] ima: Allocated hash algorithm: sha1 [ 3.205550] ima: No architecture policies found [ 3.207152] evm: Initialising EVM extended attributes: [ 3.208617] evm: security.selinux [ 3.209642] evm: security.ima [ 3.210627] evm: security.capability [ 3.211501] evm: HMAC attrs: 0x1 [ 3.213150] rtc_cmos 00:05: setting system clock to 2026-09-15 03:58:10 UTC (1789444690) [ 3.217436] debug: unmapping init [mem 0xffffffff90e03000-0xffffffff90ffffff] [ 3.219517] debug: unmapping init [mem 0xffffffff8fb82000-0xffffffff8fe58fff] [ 3.227135] Write protecting the kernel read-only data: 28672k [ 3.229364] debug: unmapping init [mem 0xffffffff8e203000-0xffffffff8e3fffff] [ 3.231172] debug: unmapping init [mem 0xffffffff8eb14000-0xffffffff8ebfffff] [ 3.257429] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.263558] systemd[1]: Detected virtualization kvm. [ 3.264885] systemd[1]: Detected architecture x86-64. [ 3.266195] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.291032] systemd[1]: No hostname configured. [ 3.292207] systemd[1]: Set hostname to . [ 3.294182] random: systemd: uninitialized urandom read (16 bytes read) [ 3.296290] systemd[1]: Initializing machine ID from random generator. [ 3.343128] random: ln: uninitialized urandom read (6 bytes read) [ 3.410463] random: systemd: uninitialized urandom read (16 bytes read) [ 3.412566] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.416887] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.419837] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console.[ 3.519579] random: fast init done [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.921193] device-mapper: uevent: version 1.0.3 [ 3.923418] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.592567] virtio_net virtio0 ens2: renamed from eth0 [ 4.678517] scsi host0: ata_piix [ 4.698862] scsi host1: ata_piix [ 4.702495] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.704473] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.282058] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.715917] random: crng init done [ 9.717094] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.958932] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.939275] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.167472] SELinux: Disabled at runtime. [ 11.229101] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.238446] systemd[1]: Detected virtualization kvm. [ 11.240108] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.674038] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.676812] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.681176] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.685372] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.690208] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.696590] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.703280] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ 11.773457] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.122940] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.397713] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.429685] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.525154] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.534290] EDAC sbridge: Ver: 1.1.2 [ 14.138777] Key type dns_resolver registered [ 14.428362] NFS: Registering the id_resolver key type [ 14.430436] Key type id_resolver registered [ 14.432011] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg234-server login: [ 43.590911] libcfs: loading out-of-tree module taints kernel. [ 43.635887] Key type ._llcrypt registered [ 43.637919] Key type .llcrypt registered [ 43.762430] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 54.384314] hrtimer: interrupt took 4476243 ns [ 68.563244] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 70.105891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.121854] alg: No test for adler32 (adler32-zlib) [ 71.808443] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 72.552562] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 74.353077] Key type lgssc registered [ 76.550209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.851556] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.145839] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 146.658612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.701403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.050616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 148.068538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.140917] Lustre: lustre-MDT0000: new disk, initializing [ 148.254198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.273086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 152.676509] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.392728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.489207] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.508804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.512602] Lustre: Skipped 1 previous similar message [ 164.579943] Lustre: lustre-MDT0001: new disk, initializing [ 164.645569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.666354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.677483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.902972] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.768863] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 183.028817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.322917] Lustre: lustre-OST0000: new disk, initializing [ 183.326570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 183.334111] Lustre: 8467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 183.396794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.877319] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.057442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.076851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.143175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.291656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.414606] Lustre: lustre-OST0001: new disk, initializing [ 204.417703] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.422326] Lustre: 9538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.477423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.948215] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.519635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 211.551332] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.595991] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 224.344174] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.631508] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.437919] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.234373] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing yml_node [ 244.261345] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 246.494042] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 249.091242] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 250.399509] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 00:02:16 EDT 2026 [ 269.810946] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 271.433932] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.226309] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.227552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.240496] Lustre: Skipped 3 previous similar messages [ 288.252657] Lustre: Skipped 3 previous similar messages [ 293.346296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.350433] Lustre: Skipped 2 previous similar messages [ 294.275382] Lustre: server umount lustre-MDT0000 complete [ 297.984742] LustreError: 9548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444985 with bad export cookie 2330990801374804949 [ 297.999584] LustreError: 9548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 298.009812] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.464991] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.481466] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.496994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.513931] Lustre: Skipped 2 previous similar messages [ 307.687139] LustreError: 12279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.692420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.710199] LustreError: 12279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 307.716201] Lustre: Skipped 1 previous similar message [ 308.706944] LustreError: 10117:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.731400] LustreError: 10117:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 312.289540] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.631311] Lustre: server umount lustre-MDT0001 complete [ 316.188265] Lustre: server umount lustre-OST0000 complete [ 319.731427] Lustre: server umount lustre-OST0001 complete [ 325.147088] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 333.730600] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 372.397607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.725762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 372.758680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 372.871890] Lustre: lustre-MDT0000: new disk, initializing [ 372.985853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.010779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 377.410429] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 389.014934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.076642] Lustre: 15096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 389.105707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 389.108874] Lustre: Skipped 1 previous similar message [ 389.156720] Lustre: lustre-MDT0001: new disk, initializing [ 389.229816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 389.260370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 389.270632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 393.528608] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 403.690792] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 409.520539] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 418.730943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.226863] Lustre: lustre-OST0000: new disk, initializing [ 419.235638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 419.257088] Lustre: 17062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 419.412217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.391220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 420.407985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 420.604629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 426.135078] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 436.257742] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 442.517376] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 442.947459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.356231] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.674591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 456.683413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.700417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.705741] LustreError: Skipped 1 previous similar message [ 456.724466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.735054] Lustre: Skipped 2 previous similar messages [ 456.762471] Lustre: Skipped 3 previous similar messages [ 459.481308] Lustre: server umount lustre-OST0000 complete [ 465.894070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 465.899884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.495704] Lustre: server umount lustre-MDT0000 complete [ 472.040183] LustreError: 17737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.060784] LustreError: 17737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 475.097882] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445162 with bad export cookie 2330990801374806475 [ 475.100891] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.114721] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 475.530161] Lustre: server umount lustre-MDT0001 complete [ 480.383316] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:06:06 (1789445166) [ 483.285326] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 484.957846] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:06:11 (1789445171) [ 492.151347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.639740] LustreError: 19476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.716127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.499565] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 498.149321] LustreError: 19477:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.361805] LustreError: 19476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.384738] LustreError: 19476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 508.045562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.788418] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 523.150152] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.307832] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 538.201873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.450813] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 554.919679] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 578.023917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.035277] Lustre: Skipped 1 previous similar message [ 578.063979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 578.072662] Lustre: Skipped 4 previous similar messages [ 580.008440] Lustre: server umount lustre-OST0000 complete [ 585.194412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 588.264378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.298927] Lustre: Skipped 2 previous similar messages [ 590.718358] Lustre: server umount lustre-MDT0000 complete [ 593.382772] LustreError: 19476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.397838] LustreError: 19476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 595.394152] LustreError: 19458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445282 with bad export cookie 2330990801374807364 [ 595.407321] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.414671] LustreError: 19458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 596.148558] Lustre: server umount lustre-MDT0001 complete [ 609.396890] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 614.730248] Key type lgssc unregistered [ 615.380643] LNet: 23031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.396415] LNetError: 23031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 616.446111] LNet: Removed LNI 192.168.202.134@tcp [ 618.210182] Key type .llcrypt unregistered [ 618.217230] Key type ._llcrypt unregistered [ 642.432314] Key type ._llcrypt registered [ 642.434061] Key type .llcrypt registered [ 642.566549] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 643.551835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 643.568710] alg: No test for adler32 (adler32-zlib) [ 644.730921] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 644.978224] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 646.664225] Key type lgssc registered [ 647.943291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 657.774529] loop: module loaded [ 657.879739] loop0: detected capacity change from 0 to 204800000 [ 658.170492] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 658.186801] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 658.601145] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.407816] loop0: detected capacity change from 0 to 204800000 [ 665.671329] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.687827] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.711321] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 671.395372] loop0: detected capacity change from 0 to 204800000 [ 678.546938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 678.595558] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.982420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 680.051227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 680.176720] Lustre: lustre-MDT0000: new disk, initializing [ 680.350982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.412659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 685.729595] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 695.494703] loop1: detected capacity change from 0 to 204800000 [ 701.946977] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 702.302579] Lustre: lustre-OST07c6: new disk, initializing [ 702.307760] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 702.311163] Lustre: Skipped 1 previous similar message [ 702.315587] Lustre: 26213:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 702.400842] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 709.479323] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 711.290237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 711.317358] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 711.372238] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 715.499339] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 715.506617] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 715.512662] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 715.520181] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 715.527212] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 715.531156] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 728.103767] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 728.115474] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 728.122549] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 728.128321] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 728.133755] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 728.138423] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 733.109529] Lustre: 25200:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 733.113845] Lustre: 25200:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 733.117862] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 733.122232] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.125414] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 733.128899] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.131695] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 733.136344] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.140202] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 733.150748] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.163877] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 733.171762] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.420146] Lustre: 25201:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 736.432805] Lustre: 25201:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 736.450739] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 736.459850] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.477158] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 736.499990] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.512109] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 736.534549] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.545795] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 736.553405] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.560102] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 736.570877] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.810084] Lustre: 25200:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 741.820631] Lustre: 25200:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 741.825309] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 741.830100] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.840436] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 741.850647] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.861051] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 741.874132] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.887430] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 741.896983] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.908803] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 741.915141] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.133917] Lustre: 25200:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 750.140634] Lustre: 25200:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 750.144959] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.150029] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 750.154930] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.159187] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 750.163065] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 750.167722] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 750.170196] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 750.173444] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 750.177373] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.180919] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 803.297502] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 803.308923] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.331292] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 808.420773] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 808.692332] Lustre: server umount lustre-OST07c6 complete [ 819.326775] Lustre: server umount lustre-MDT0000 complete [ 837.818805] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 840.688957] Key type lgssc unregistered [ 840.968686] LNet: 28769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.974657] LNetError: 28769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.031768] LNet: Removed LNI 192.168.202.134@tcp [ 843.290312] Key type .llcrypt unregistered [ 843.293590] Key type ._llcrypt unregistered [ 862.036367] Key type ._llcrypt registered [ 862.038296] Key type .llcrypt registered [ 862.131749] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 862.956839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 863.003773] alg: No test for adler32 (adler32-zlib) [ 864.108852] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 864.335722] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 866.048545] Key type lgssc registered [ 867.289958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 875.810401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 875.824341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.239731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.281213] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 886.769643] LustreError: 30037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 886.791714] LustreError: 30037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 891.882116] LustreError: 30036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.118676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.733488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 899.191782] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 907.415527] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 914.062920] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 925.460435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.245648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.116589] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 945.824984] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 946.771076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 964.068116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 964.090397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.110290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.913313] Lustre: server umount lustre-OST0000 complete [ 970.743276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 970.751931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.782814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 972.261942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.265145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 972.297404] Lustre: Skipped 1 previous similar message [ 977.379404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.402289] Lustre: Skipped 1 previous similar message [ 979.521141] Lustre: server umount lustre-MDT0000 complete [ 982.499920] LustreError: 30036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 982.542403] LustreError: 30036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 983.647920] LustreError: 30017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445670 with bad export cookie 2628454314220493669 [ 983.658716] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 983.665790] LustreError: 30017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 984.416978] Lustre: server umount lustre-MDT0001 complete [ 998.246926] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1001.934675] Key type lgssc unregistered [ 1002.230239] LNet: 33779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.237113] LNetError: 33779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.256400] LNet: Removed LNI 192.168.202.134@tcp [ 1003.088931] Key type .llcrypt unregistered [ 1003.091271] Key type ._llcrypt unregistered [ 1025.777553] Key type ._llcrypt registered [ 1025.779103] Key type .llcrypt registered [ 1025.855904] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1026.693263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.745473] alg: No test for adler32 (adler32-zlib) [ 1027.770089] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 1027.943221] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1029.585935] Key type lgssc registered [ 1031.061739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1041.966060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1041.979625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.589507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.134460] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1053.157375] LustreError: 35047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.170732] LustreError: 35047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1058.288895] LustreError: 35046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.916082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.308245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1065.412099] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1074.116624] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1080.513694] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1088.504977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1088.874814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.967704] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1105.264444] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1113.474970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1128.932852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.939512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.949217] Lustre: Skipped 1 previous similar message [ 1128.955890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.975794] Lustre: Skipped 1 previous similar message [ 1132.972443] Lustre: server umount lustre-OST0000 complete [ 1137.121724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1137.127173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.139257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.172436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.176264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.205499] Lustre: Skipped 1 previous similar message [ 1143.514772] Lustre: server umount lustre-MDT0000 complete [ 1144.295264] LustreError: 35047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.337232] LustreError: 35047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1147.908270] LustreError: 36892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445835 with bad export cookie 11141989161227639609 [ 1147.913110] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.923776] LustreError: 36892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1148.387967] Lustre: server umount lustre-MDT0001 complete [ 1160.299729] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1162.971953] Key type lgssc unregistered [ 1163.273436] LNet: 38605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.278734] LNetError: 38605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.315096] LNet: Removed LNI 192.168.202.134@tcp [ 1164.157551] Key type .llcrypt unregistered [ 1164.167073] Key type ._llcrypt unregistered [ 1188.517158] Key type ._llcrypt registered [ 1188.520758] Key type .llcrypt registered [ 1188.618680] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1189.479903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1189.642739] alg: No test for adler32 (adler32-zlib) [ 1190.756402] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 1190.984913] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1192.656457] Key type lgssc registered [ 1193.668722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1201.017254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1201.045933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.596138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.147356] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1211.874864] LustreError: 40021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1211.890202] LustreError: 40021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1216.996551] LustreError: 40022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.476278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.041418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.015436] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1232.255188] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1238.838310] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1246.244506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.520843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.541793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1253.900202] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1262.858387] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1269.847977] Lustre: server umount lustre-MDT0000 complete [ 1270.523066] LustreError: 41009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.318438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.321899] LustreError: 41009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.339154] Lustre: Skipped 2 previous similar messages [ 1273.352946] LustreError: 41009:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1273.368636] LustreError: 40001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445960 with bad export cookie 10088329263130259394 [ 1273.378812] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.603254] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [ 1278.433509] LustreError: 40017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.437232] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.451416] LustreError: 40017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1278.464028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1279.762945] Lustre: server umount lustre-MDT0001 complete [ 1309.152161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1309.562922] Lustre: server umount lustre-OST0000 complete [ 1329.065475] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1331.989488] Key type lgssc unregistered [ 1332.293331] LNet: 43885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.306783] LNetError: 43885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.334801] LNet: Removed LNI 192.168.202.134@tcp [ 1333.166431] Key type .llcrypt unregistered [ 1333.168123] Key type ._llcrypt unregistered [ 1359.604455] Key type ._llcrypt registered [ 1359.605847] Key type .llcrypt registered [ 1359.717952] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1361.629441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.675786] alg: No test for adler32 (adler32-zlib) [ 1362.871794] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 1363.218085] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1364.970744] Key type lgssc registered [ 1366.626410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1375.547074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1375.584941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1377.096600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.676624] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1387.495694] LustreError: 45300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1387.519741] LustreError: 45300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1392.609599] LustreError: 45299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.411953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.911796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1401.407923] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1412.685954] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1420.754254] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1430.599988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1431.203703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.756674] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1447.558887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1448.796635] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1456.907221] Lustre: lustre-OST0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 1458.147386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1458.153355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.201474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1458.212943] Lustre: Skipped 1 previous similar message [ 1461.960856] Lustre: lustre-OST0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 1462.684892] Lustre: server umount lustre-OST0000 complete [ 1482.209791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1482.214544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.221106] Lustre: Skipped 1 previous similar message [ 1482.224662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1483.754616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.826397] Lustre: server umount lustre-MDT0000 complete [ 1488.867817] LustreError: 45300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.903309] LustreError: 45300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1491.682768] LustreError: 45279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446178 with bad export cookie 17992891958868633742 [ 1491.698612] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1498.117932] Lustre: server umount lustre-MDT0001 complete [ 1508.960389] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1511.979775] Key type lgssc unregistered [ 1512.277245] LNet: 49009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.282978] LNetError: 49009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.298089] LNet: Removed LNI 192.168.202.134@tcp [ 1513.089681] Key type .llcrypt unregistered [ 1513.093410] Key type ._llcrypt unregistered [ 1539.564683] Key type ._llcrypt registered [ 1539.566587] Key type .llcrypt registered [ 1539.756693] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1541.144549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1541.286815] alg: No test for adler32 (adler32-zlib) [ 1542.657607] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 1543.055769] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1544.824269] Key type lgssc registered [ 1546.225158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1555.791580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1555.809978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.505752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.887964] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1567.713666] LustreError: 50424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1567.754145] LustreError: 50424:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1572.164868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.478324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1578.133366] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1587.075325] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1592.977626] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1600.174794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1600.568570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.649322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1607.715854] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1616.981904] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1622.765152] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1627.650691] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1640.937934] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-15 0h25m25s [ 1643.161405] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1648.103594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.112510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.123156] Lustre: Skipped 2 previous similar messages [ 1648.148146] Lustre: Skipped 2 previous similar messages [ 1649.188352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.437833] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 1650.446672] Lustre: Skipped 3 previous similar messages [ 1652.024675] Lustre: server umount lustre-MDT0000 complete [ 1655.268537] LustreError: 50424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.303829] LustreError: 50424:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1656.537820] LustreError: 51394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.024603] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1661.420497] LustreError: 50420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.450704] LustreError: 50420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1665.580349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.730608] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1666.026311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.124830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1667.075691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1671.183431] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1674.613978] Lustre: DEBUG MARKER: Wait for df (36669) ... [ 1677.344619] Lustre: DEBUG MARKER: done [ 1683.947385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1683.969604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.996716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1686.499514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1686.516064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.894623] Lustre: server umount lustre-OST0000 complete [ 1697.761724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.763403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1697.788510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.794573] Lustre: Skipped 4 previous similar messages [ 1701.162153] Lustre: server umount lustre-MDT0000 complete [ 1704.959825] LustreError: 51152:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.970674] LustreError: 51152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1706.081108] LustreError: 50406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446393 with bad export cookie 11773039056485466555 [ 1706.085184] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1706.091417] LustreError: 50406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1706.483197] Lustre: server umount lustre-MDT0001 complete [ 1718.512755] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1721.384404] Key type lgssc unregistered [ 1721.762024] LNet: 55350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.774299] LNetError: 55350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.860509] LNet: Removed LNI 192.168.202.134@tcp [ 1724.356847] Key type .llcrypt unregistered [ 1724.358729] Key type ._llcrypt unregistered [ 1771.254690] Key type ._llcrypt registered [ 1771.258367] Key type .llcrypt registered [ 1771.351822] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1772.331479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1772.382477] alg: No test for adler32 (adler32-zlib) [ 1773.811512] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 1774.071646] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1775.800278] Key type lgssc registered [ 1777.275524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1785.844425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1785.870386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.331679] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1787.403706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1787.694905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.201718] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1803.031908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.144346] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1803.168500] Lustre: 57557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1803.534396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1808.657605] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1816.446294] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1822.673988] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1830.711821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1830.937673] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1831.123712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.039308] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1840.862029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1846.279790] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1854.275648] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-15 0h28m59s [ 1856.247536] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1863.613108] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1866.557573] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1866.647089] LustreError: 57566:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1886.966261] Lustre: lustre-MDT0000: Client 9ea46e59-7bca-4da4-92ec-1fdbc4afdc69 (at 192.168.202.34@tcp) reconnecting [ 1906.648194] LustreError: 57566:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1910.765519] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1914.249683] Lustre: DEBUG MARKER: done [ 1923.049517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.057622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.074346] Lustre: Skipped 1 previous similar message [ 1923.083596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1923.091753] Lustre: Skipped 1 previous similar message [ 1928.265592] Lustre: server umount lustre-OST0000 complete [ 1933.282133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.299761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1933.305219] Lustre: Skipped 1 previous similar message [ 1934.369503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.382344] Lustre: Skipped 1 previous similar message [ 1939.427533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.439079] Lustre: Skipped 1 previous similar message [ 1945.574252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.589417] Lustre: Skipped 1 previous similar message [ 1947.104604] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1947.443589] Lustre: server umount lustre-MDT0000 complete [ 1951.723789] LustreError: 57570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.738893] LustreError: 57570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1952.103159] LustreError: 60319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446639 with bad export cookie 18079786812832939505 [ 1952.115955] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.786193] Lustre: server umount lustre-MDT0001 complete [ 1965.945830] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1970.236711] Key type lgssc unregistered [ 1970.594704] LNet: 61890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.600908] LNetError: 61890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1970.618510] LNet: Removed LNI 192.168.202.134@tcp [ 1971.868939] Key type .llcrypt unregistered [ 1971.873574] Key type ._llcrypt unregistered [ 2023.651213] Key type ._llcrypt registered [ 2023.655890] Key type .llcrypt registered [ 2023.770257] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2025.459085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2025.635927] alg: No test for adler32 (adler32-zlib) [ 2026.724858] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 2027.049962] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2028.787295] Key type lgssc registered [ 2030.029985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2034.159454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2034.185843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.602931] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2035.646770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2035.824217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.886594] Lustre: Failing over lustre-MDT0000 [ 2037.076555] Lustre: server umount lustre-MDT0000 complete [ 2044.459641] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:32:10 (1789446730) [ 2052.487726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2052.987338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.437540] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2069.258219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.543228] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2069.647220] Lustre: 64686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2069.962200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2075.131579] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2085.796452] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2092.161928] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2101.308921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2101.538093] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2101.782748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2109.965573] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2111.240946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2119.620332] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2126.568173] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2136.548489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2136.570729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.587155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.082468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.097290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.113895] Lustre: Skipped 1 previous similar message [ 2137.126249] Lustre: Skipped 1 previous similar message [ 2141.588497] Lustre: server umount lustre-MDT0000 complete [ 2145.728685] LustreError: 65433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446833 with bad export cookie 13229923826869181237 [ 2145.734710] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.738032] LustreError: 65433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2146.385809] Lustre: server umount lustre-MDT0001 complete [ 2150.132983] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2157.187802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2163.616167] Lustre: 63610:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789446834/real 1789446834] req@0000000086ad31c8 x1876370877623168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789446850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.657490] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.729974] Lustre: 63608:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789446837/real 1789446837] req@0000000030961db4 x1876370877623296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789446853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2167.673169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.779461] Lustre: 63607:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789446839/real 1789446839] req@000000005bdf5002 x1876370877623424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789446855 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2171.258888] LustreError: 66538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.278603] LustreError: 66538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2171.372491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.527087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2172.322195] Lustre: 63610:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789446843/real 1789446843] req@000000001ecaca88 x1876370877623680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789446859 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2173.411088] LustreError: 68461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.433852] LustreError: 68461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2176.521275] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2177.507797] Lustre: 63609:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789446848/real 1789446848] req@00000000083d2f2c x1876370877623808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789446864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2178.530738] LustreError: 66539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.555519] LustreError: 66539:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2183.653103] LustreError: 68459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.687818] LustreError: 68459:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2185.998691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.376727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2191.857287] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2192.942604] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2193.170511] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2202.751749] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2210.599153] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2222.049808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2222.055962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.077331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2225.537022] Lustre: server umount lustre-MDT0000 complete [ 2228.726527] LustreError: 69948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.759926] LustreError: 69948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2230.148588] LustreError: 64678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446917 with bad export cookie 13229923826869183512 [ 2230.154856] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2230.167778] LustreError: 64678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2230.709161] Lustre: server umount lustre-MDT0001 complete [ 2239.359642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2250.211169] Lustre: 63610:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789446921/real 1789446921] req@000000003cbb6d0c x1876370877654400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789446937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2250.251762] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.265485] Lustre: Skipped 2 previous similar messages [ 2250.972815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.395848] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb79a21a01a03a97b [ 2255.401699] Lustre: MGC192.168.202.134@tcp: Connection restored to 0@lo (at 0@lo) [ 2255.639722] LustreError: 66539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.676860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.736625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2260.654871] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2269.528223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2274.790551] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2275.315707] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2279.422607] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2282.487918] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2287.927681] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2295.779991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2295.789973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.804567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.810615] Lustre: Skipped 3 previous similar messages [ 2299.874758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.095592] Lustre: server umount lustre-MDT0000 complete [ 2304.997928] LustreError: 72155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.008325] LustreError: 72155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2305.865677] LustreError: 64680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446993 with bad export cookie 13229923826869184891 [ 2305.873877] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2305.886098] LustreError: 64680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2306.250938] Lustre: server umount lustre-MDT0001 complete [ 2313.111815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2316.780305] Lustre: DEBUG MARKER: files compared the same [ 2329.959239] Lustre: server umount lustre-OST0000 complete [ 2348.313670] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2351.476656] Key type lgssc unregistered [ 2351.804579] LNet: 74420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.814933] LNetError: 74420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.838102] LNet: Removed LNI 192.168.202.134@tcp [ 2352.670592] Key type .llcrypt unregistered [ 2352.673145] Key type ._llcrypt unregistered [ 2379.699178] Key type ._llcrypt registered [ 2379.707947] Key type .llcrypt registered [ 2379.861811] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2381.820257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2381.838583] alg: No test for adler32 (adler32-zlib) [ 2383.070490] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 2383.461828] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2385.232719] Key type lgssc registered [ 2386.149899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2395.066624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2395.117364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.916220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.536381] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug +malloc all [ 2407.397228] LustreError: 75834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.423549] LustreError: 75834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2412.524201] LustreError: 75835:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.200692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.709082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2419.913931] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug +malloc all [ 2428.490388] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2434.457212] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2442.049524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2442.496840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.511288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2450.170158] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug +malloc all [ 2460.488574] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2474.473992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.475499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2474.478625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2474.491166] Lustre: Skipped 1 previous similar message [ 2476.293295] Lustre: server umount lustre-OST0000 complete [ 2481.121332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2481.128279] LustreError: Skipped 1 previous similar message [ 2481.132208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.147509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2481.160846] Lustre: Skipped 1 previous similar message [ 2484.705690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2484.711884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.400210] Lustre: server umount lustre-MDT0000 complete [ 2489.826641] LustreError: 75834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.849911] LustreError: 75834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2491.062717] LustreError: 75814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447178 with bad export cookie 15642753013073817547 [ 2491.073162] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.086406] LustreError: 75814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2491.662829] Lustre: server umount lustre-MDT0001 complete [ 2506.768878] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2508.877574] Key type lgssc unregistered [ 2509.150781] LNet: 79365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.172792] LNetError: 79365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.249515] LNet: Removed LNI 192.168.202.134@tcp [ 2510.951398] Key type .llcrypt unregistered [ 2510.957398] Key type ._llcrypt unregistered [ 2537.302621] Key type ._llcrypt registered [ 2537.305201] Key type .llcrypt registered [ 2537.450533] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2538.870185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2538.903402] alg: No test for adler32 (adler32-zlib) [ 2540.019847] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 2540.446662] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2542.200255] Key type lgssc registered [ 2543.622952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.115347] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2551.155373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2566.688932] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2571.808558] LustreError: 80752:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 2598.625141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.811160] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2615.040402] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2623.175619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.322852] Lustre: *** cfs_fail_loc=706, val=0*** [ 2623.787555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.875135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2627.997106] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2629.094405] LustreError: 80777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.112064] LustreError: 80777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2633.188818] LustreError: 81838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.236606] LustreError: 81838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2635.743285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.966436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2640.209205] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2648.724804] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2655.168572] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2659.809442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2659.822109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.842166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.858367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.864482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2661.876272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.448331] Lustre: server umount lustre-OST0000 complete [ 2666.979318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2666.984451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.990574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2666.991861] Lustre: Skipped 1 previous similar message [ 2667.007754] Lustre: Skipped 2 previous similar messages [ 2672.097272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.101943] Lustre: Skipped 1 previous similar message [ 2677.221506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.226127] Lustre: Skipped 1 previous similar message [ 2681.313085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2681.540746] Lustre: server umount lustre-MDT0000 complete [ 2682.342640] LustreError: 81837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.388336] LustreError: 81837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2686.518786] LustreError: 80760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447373 with bad export cookie 13240855632306103817 [ 2686.527982] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2686.541466] LustreError: 80760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2687.039684] Lustre: server umount lustre-MDT0001 complete [ 2699.056352] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2701.541262] Key type lgssc unregistered [ 2701.853106] LNet: 84367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.863268] LNetError: 84367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2701.880244] LNet: Removed LNI 192.168.202.134@tcp [ 2702.707435] Key type .llcrypt unregistered [ 2702.708967] Key type ._llcrypt unregistered [ 2729.502548] Key type ._llcrypt registered [ 2729.505922] Key type .llcrypt registered [ 2729.640574] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2730.467694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2730.476658] alg: No test for adler32 (adler32-zlib) [ 2731.550197] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 2731.801562] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2733.480438] Key type lgssc registered [ 2734.620584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2743.360814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2743.394540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.285363] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2757.311127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2757.889019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.979641] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2763.240375] LustreError: 86469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.263269] LustreError: 86469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2768.358658] LustreError: 86468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.377517] LustreError: 86468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2770.562749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2770.859510] LustreError: 86469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.897249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.033813] LustreError: 86473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.047190] LustreError: 86473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2777.222435] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2781.153637] LustreError: 86472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.171535] LustreError: 86472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2784.653034] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2784.661652] Lustre: MGS: Client 11fbaf66-cb07-4385-b07b-77ded8a2ff65 (at 0@lo) reconnecting [ 2784.676738] LustreError: 86483:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2784.817781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2784.882520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2788.771204] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2802.152785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.160590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.173163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.219485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.235283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2805.253760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.080423] Lustre: server umount lustre-OST0000 complete [ 2815.457927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.463845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2815.481124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.489271] Lustre: Skipped 2 previous similar messages [ 2818.323692] Lustre: server umount lustre-MDT0000 complete [ 2820.594216] LustreError: 86467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.623047] LustreError: 86467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2822.594102] Lustre: server umount lustre-MDT0001 complete [ 2826.648235] Lustre: server umount MGS complete [ 2838.387617] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:45:24 (1789447524) [ 2864.379469] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2876.800205] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2912.901506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.732387] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2927.028985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.356186] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2927.410312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2927.499666] Lustre: lustre-MDT0001: new disk, initializing [ 2927.563712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2927.582991] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2927.597340] Lustre: 93637:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2927.607524] LustreError: 93637:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2931.921178] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2942.110497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2942.287245] Lustre: lustre-OST0000: new disk, initializing [ 2942.289514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2942.295967] Lustre: 94411:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2942.351938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.176183] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2952.300269] Lustre: 94881:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002094d0e5 x1876371616865664/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789447655 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2957.602265] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2964.297465] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.315273] Lustre: MGS: Client 338a1f3f-3612-4f9e-9491-b160a7f783dc (at 0@lo) reconnecting [ 2964.382580] Lustre: 95456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2964.401493] LustreError: 93623:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2964.430188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2964.585120] Lustre: lustre-MDT0000: new disk, initializing [ 2964.756288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.792686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2964.911200] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2965.864184] Lustre: 95481:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d7b8ff48 x1876371616872192/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789447669 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2969.708379] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2970.920099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2970.926591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2971.139929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2995.684917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.688201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.703138] Lustre: Skipped 2 previous similar messages [ 2995.711793] Lustre: Skipped 1 previous similar message [ 3000.806199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.823149] Lustre: Skipped 1 previous similar message [ 3001.274574] Lustre: server umount lustre-OST0000 complete [ 3005.952236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.960192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3011.044816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.056053] Lustre: Skipped 3 previous similar messages [ 3011.361922] Lustre: server umount lustre-MDT0000 complete [ 3022.355422] Lustre: server umount lustre-MDT0001 complete [ 3026.643667] Lustre: server umount MGS complete [ 3039.842776] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:48:45 (1789447725) [ 3060.017340] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3062.705809] Key type lgssc unregistered [ 3063.115330] LNet: 97979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3063.123061] LNetError: 97979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3063.163047] LNet: Removed LNI 192.168.202.134@tcp [ 3063.993241] Key type .llcrypt unregistered [ 3063.995521] Key type ._llcrypt unregistered [ 3081.311355] Key type ._llcrypt registered [ 3081.315831] Key type .llcrypt registered [ 3081.435822] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3082.535917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3082.687321] alg: No test for adler32 (adler32-zlib) [ 3083.892252] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 3084.206256] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3085.880948] Key type lgssc registered [ 3087.107720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3093.731537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3093.756857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.097539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.689359] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3105.268871] LustreError: 99199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3105.304549] LustreError: 99199:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3109.622251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.122486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3115.978131] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3124.991535] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3130.602677] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3136.534658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3136.782363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.932904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3142.869809] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3150.686631] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3165.676439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3165.685595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.690050] LustreError: Skipped 1 previous similar message [ 3165.692055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3165.707606] Lustre: Skipped 1 previous similar message [ 3165.720156] Lustre: Skipped 1 previous similar message [ 3168.063163] Lustre: server umount lustre-OST0000 complete [ 3171.810799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3171.824677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.835287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3175.911280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3175.917752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.929810] Lustre: Skipped 1 previous similar message [ 3177.802351] Lustre: server umount lustre-MDT0000 complete [ 3181.021472] LustreError: 99178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447868 with bad export cookie 14511695859013489133 [ 3181.024712] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3181.030428] LustreError: 99178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3181.228831] Lustre: server umount lustre-MDT0001 complete [ 3191.448813] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3193.993275] Key type lgssc unregistered [ 3194.228518] LNet: 102729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3194.236347] LNetError: 102729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3194.246948] LNet: Removed LNI 192.168.202.134@tcp [ 3194.881348] Key type .llcrypt unregistered [ 3194.884053] Key type ._llcrypt unregistered [ 3210.040290] Key type ._llcrypt registered [ 3210.041736] Key type .llcrypt registered [ 3210.174899] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3211.214260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3211.316208] alg: No test for adler32 (adler32-zlib) [ 3212.456793] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 3212.746069] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3214.401709] Key type lgssc registered [ 3215.683763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3222.020042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3222.035339] LustreError: 103792:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3222.119519] LustreError: 103808:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3222.130844] LustreError: 103792:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4920 [ 3222.141396] LustreError: 103792:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3222.151032] LustreError: 103792:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3222.154835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.752582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3230.873988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.186547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3235.441077] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3240.087531] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing lsmod [ 3242.318077] LustreError: 105011:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3242.332962] LustreError: 105015:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3242.339956] LustreError: 105011:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3242.342691] LustreError: 105011:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3242.349590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3242.350022] LustreError: 105011:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3242.648780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3246.563043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3246.577249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.594368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.097185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.100700] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.106347] Lustre: Skipped 1 previous similar message [ 3248.126386] Lustre: Skipped 1 previous similar message [ 3253.230849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3253.245591] Lustre: Skipped 3 previous similar messages [ 3256.292608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.360151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3259.556889] Lustre: server umount lustre-MDT0000 complete [ 3261.411051] LustreError: 103846:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.424588] LustreError: 103846:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3263.293666] LustreError: 103824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447950 with bad export cookie 18093495144232816691 [ 3263.304834] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3263.309776] LustreError: 103824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3264.544331] LustreError: 104277:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002f4b7fef x1876372121211776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3264.694522] Lustre: server umount lustre-MDT0001 complete [ 3277.845849] Lustre: server umount lustre-OST0000 complete [ 3284.474490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3284.729645] LustreError: 106079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.781264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3288.926913] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3290.082620] LustreError: 106080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3295.205981] LustreError: 106079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.967818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.788235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3303.017233] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3310.702874] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3316.292549] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3325.637962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3326.102616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.241731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3333.670977] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3341.393296] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3351.012395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3351.026802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.060890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3351.082399] Lustre: Skipped 2 previous similar messages [ 3353.057108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3353.076700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.748811] Lustre: server umount lustre-OST0000 complete [ 3363.299872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.312519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.316238] Lustre: Skipped 3 previous similar messages [ 3366.645622] Lustre: server umount lustre-MDT0000 complete [ 3368.420744] LustreError: 106075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.443493] LustreError: 106075:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3369.920498] LustreError: 106060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448057 with bad export cookie 18093495144232817391 [ 3369.939961] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3370.196658] Lustre: server umount lustre-MDT0001 complete [ 3381.918701] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3385.104787] Key type lgssc unregistered [ 3385.510599] LNet: 109613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3385.521799] LNetError: 109613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3385.555490] LNet: Removed LNI 192.168.202.134@tcp [ 3386.533162] Key type .llcrypt unregistered [ 3386.539665] Key type ._llcrypt unregistered [ 3411.144299] Key type ._llcrypt registered [ 3411.146379] Key type .llcrypt registered [ 3411.257434] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3412.338236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3412.391306] alg: No test for adler32 (adler32-zlib) [ 3413.326388] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 3413.600259] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3415.235389] Key type lgssc registered [ 3416.243400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3425.428366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3425.465820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.063586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.412361] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3437.547367] LustreError: 111027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.575765] LustreError: 111027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3441.837962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.309755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3447.517590] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3455.896908] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3462.391435] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3470.744714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.242419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.275490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3478.660692] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3486.960393] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3493.022561] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3496.936319] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3503.076883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3503.079682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.087283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.111848] Lustre: Skipped 1 previous similar message [ 3507.403712] Lustre: server umount lustre-OST0000 complete [ 3513.319245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.319245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.323439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.348354] Lustre: Skipped 2 previous similar messages [ 3518.413156] Lustre: server umount lustre-MDT0000 complete [ 3518.447119] LustreError: 111028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.476129] LustreError: 111028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3522.942729] LustreError: 111007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448210 with bad export cookie 13991137788808127138 [ 3522.944369] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3522.970757] LustreError: 111007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3523.581965] Lustre: server umount lustre-MDT0001 complete [ 3534.804170] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3538.079193] Key type lgssc unregistered [ 3538.319096] LNet: 114667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3538.329144] LNetError: 114667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3538.348760] LNet: Removed LNI 192.168.202.134@tcp [ 3539.162992] Key type .llcrypt unregistered [ 3539.169030] Key type ._llcrypt unregistered [ 3560.158993] Key type ._llcrypt registered [ 3560.160939] Key type .llcrypt registered [ 3560.315547] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3561.130522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3561.311500] alg: No test for adler32 (adler32-zlib) [ 3562.726316] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 3563.006255] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3564.659667] Key type lgssc registered [ 3566.520960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3576.098747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3576.150873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.752571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.290737] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3588.080738] LustreError: 115886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.104158] LustreError: 115886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3591.586259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.849348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3597.121989] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3606.009901] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3611.468869] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3618.130868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3618.555982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.668075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3625.506116] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3633.298686] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3644.385964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3644.396669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.435308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3647.458609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3647.462128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.308721] Lustre: server umount lustre-OST0000 complete [ 3657.698763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.704982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.721309] Lustre: Skipped 1 previous similar message [ 3657.742788] Lustre: Skipped 2 previous similar messages [ 3661.045544] Lustre: server umount lustre-MDT0000 complete [ 3662.825722] LustreError: 118380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3662.838988] LustreError: 118380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3665.664435] LustreError: 115911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448352 with bad export cookie 7906693782917657417 [ 3665.670270] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3665.678301] LustreError: 115911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3666.069185] Lustre: server umount lustre-MDT0001 complete [ 3679.103844] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3682.111025] Key type lgssc unregistered [ 3682.299761] LNet: 119420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3682.309789] LNetError: 119420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3682.326490] LNet: Removed LNI 192.168.202.134@tcp [ 3683.127213] Key type .llcrypt unregistered [ 3683.129516] Key type ._llcrypt unregistered [ 3707.095257] Key type ._llcrypt registered [ 3707.097396] Key type .llcrypt registered [ 3707.252786] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3708.423653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3708.439461] alg: No test for adler32 (adler32-zlib) [ 3709.564895] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 3709.794279] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3711.432355] Key type lgssc registered [ 3712.382583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3721.292540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3721.318941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.699289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.936483] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3731.941899] LustreError: 120834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3731.959127] LustreError: 120834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3736.342519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3736.839485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3742.686437] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3751.275679] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3758.173480] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3765.915211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3766.505444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3767.564631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3774.185587] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3783.026793] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3790.562590] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3794.511795] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3794.520947] Lustre: Skipped 2 previous similar messages [ 3797.697914] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3797.747913] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3799.597421] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3799.606795] Lustre: Skipped 2 previous similar messages [ 3803.828779] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3803.832995] Lustre: Skipped 1 previous similar message [ 3811.307764] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.34@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3811.316440] Lustre: Skipped 2 previous similar messages [ 3818.060142] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.34@tcp 192.168.0.[2,10]@tcp [ 3822.500166] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.34/24@tcp in log lustre-MDT0000 [ 3822.505773] Lustre: Skipped 2 previous similar messages [ 3824.787551] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.34/24@tcp [ 3824.818760] Lustre: Skipped 1 previous similar message [ 3837.411248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.427110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3837.436031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3837.447408] Lustre: Skipped 1 previous similar message [ 3837.735052] Lustre: server umount lustre-OST0000 complete [ 3842.530995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.543506] Lustre: Skipped 1 previous similar message [ 3842.547282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3842.550868] Lustre: Skipped 1 previous similar message [ 3847.522536] Lustre: server umount lustre-MDT0000 complete [ 3847.654473] LustreError: 121785:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3847.667316] LustreError: 121785:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3850.761029] LustreError: 122677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448538 with bad export cookie 15106888623835574959 [ 3850.766202] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3850.767406] LustreError: 122677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3851.066503] Lustre: server umount lustre-MDT0001 complete [ 3861.153645] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3864.571250] Key type lgssc unregistered [ 3865.046121] LNet: 125692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3865.067429] LNetError: 125692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3865.089870] LNet: Removed LNI 192.168.202.134@tcp [ 3866.132030] Key type .llcrypt unregistered [ 3866.133460] Key type ._llcrypt unregistered [ 3895.837928] Key type ._llcrypt registered [ 3895.840639] Key type .llcrypt registered [ 3896.034270] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3897.562322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3897.690505] alg: No test for adler32 (adler32-zlib) [ 3898.931909] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 3899.390956] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3901.136815] Key type lgssc registered [ 3902.980307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3913.167794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3913.190097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.675137] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3914.706884] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.34/24@tcp [ 3914.728975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.888700] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3923.945145] LustreError: 127257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3923.958684] LustreError: 127257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3929.062656] LustreError: 127276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.887837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.091740] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3930.124197] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.34/24@tcp [ 3930.170728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3935.227201] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3942.487874] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3948.357680] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3955.589077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.120261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.257879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3963.611518] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3971.989120] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3985.891039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3985.900304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.902275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3985.927669] Lustre: Skipped 1 previous similar message [ 3991.012399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3991.017238] Lustre: Skipped 2 previous similar messages [ 3991.557516] Lustre: server umount lustre-OST0000 complete [ 3996.133600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.156607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3996.171968] Lustre: Skipped 1 previous similar message [ 4001.252219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.258342] Lustre: Skipped 1 previous similar message [ 4002.026404] Lustre: server umount lustre-MDT0000 complete [ 4005.820085] LustreError: 127238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448693 with bad export cookie 12051492860688835750 [ 4005.823142] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.839632] LustreError: 127238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4006.392766] Lustre: server umount lustre-MDT0001 complete [ 4020.914407] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4024.732584] Key type lgssc unregistered [ 4024.974632] LNet: 130848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4024.978646] LNetError: 130848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4026.053097] LNet: Removed LNI 192.168.202.134@tcp [ 4027.217381] Key type .llcrypt unregistered [ 4027.221329] Key type ._llcrypt unregistered [ 4057.852570] Key type ._llcrypt registered [ 4057.854852] Key type .llcrypt registered [ 4057.959643] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4072.354452] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4073.753740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4073.879643] alg: No test for adler32 (adler32-zlib) [ 4075.187855] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 4075.454404] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4077.152337] Key type lgssc registered [ 4078.492625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4117.771086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4117.794113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4119.101268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4119.121397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4119.178486] Lustre: lustre-MDT0000: new disk, initializing [ 4119.242932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4119.261349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4123.750441] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4133.915131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.049134] Lustre: 134247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4134.118782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4134.122306] Lustre: Skipped 1 previous similar message [ 4134.201155] Lustre: lustre-MDT0001: new disk, initializing [ 4134.335920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4134.374056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4134.393889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4140.192742] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4149.921709] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4155.722529] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4162.489330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4162.804848] Lustre: lustre-OST0000: new disk, initializing [ 4162.808612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4162.817906] Lustre: 136215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4162.918895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.541374] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4168.747509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4168.762838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4168.858357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4178.241179] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4182.909572] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4183.073070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4187.904964] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4188.125523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4194.789380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.790597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4194.811427] Lustre: Skipped 1 previous similar message [ 4194.824965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.828265] LustreError: Skipped 1 previous similar message [ 4194.846964] Lustre: Skipped 1 previous similar message [ 4196.212524] Lustre: server umount lustre-OST0000 complete [ 4200.936873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4200.947885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.974200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.030522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.031712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.194398] Lustre: server umount lustre-MDT0000 complete [ 4212.097959] LustreError: 134239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448899 with bad export cookie 13565422815795346519 [ 4212.109519] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4212.110447] LustreError: 134239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4212.474567] Lustre: server umount lustre-MDT0001 complete [ 4222.146511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4222.872116] LustreError: 138433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.896671] LustreError: 138433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4222.956482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4228.072143] LustreError: 138434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.301362] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4233.188984] LustreError: 138433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.944383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.317021] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4238.332694] Lustre: Skipped 1 previous similar message [ 4238.581456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4243.952858] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4253.147643] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4259.950413] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4267.677039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4267.998124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.785902] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4284.297848] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4290.461722] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4290.729526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4295.793188] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4295.982092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4304.684969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4304.892389] Lustre: lustre-OST0001: new disk, initializing [ 4304.896200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4304.901672] Lustre: 141806:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4305.011810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4312.235452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4312.244843] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4312.442591] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4312.772642] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4321.454179] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4321.722739] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4326.785974] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4327.004971] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4332.904426] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fa906c74800.ost_server_uuid 50 [ 4334.507765] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fa906c74800.ost_server_uuid in IDLE state after 0 sec [ 4343.272182] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.277868] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4343.279506] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4343.283151] Lustre: Skipped 1 previous similar message [ 4348.220182] Lustre: server umount lustre-OST0001 complete [ 4348.395953] LustreError: 141752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.424079] LustreError: 141752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4353.510399] LustreError: 140256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.514377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.519193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.519204] Lustre: Skipped 1 previous similar message [ 4353.549184] LustreError: 140256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4353.569958] Lustre: Skipped 1 previous similar message [ 4358.520642] Lustre: server umount lustre-OST0000 complete [ 4363.745385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.373494] Lustre: server umount lustre-MDT0000 complete [ 4368.870268] LustreError: 142641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.903101] LustreError: 142641:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4372.814444] LustreError: 140276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449060 with bad export cookie 13565422815795347394 [ 4372.825801] LustreError: 140276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4372.830257] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4373.112724] Lustre: server umount lustre-MDT0001 complete [ 4394.503522] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4397.681828] Key type lgssc unregistered [ 4398.086935] LNet: 144810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4398.101672] LNetError: 144810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4398.122395] LNet: Removed LNI 192.168.202.134@tcp [ 4399.121974] Key type .llcrypt unregistered [ 4399.131411] Key type ._llcrypt unregistered [ 4441.329165] Key type ._llcrypt registered [ 4441.334791] Key type .llcrypt registered [ 4441.433960] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4455.413184] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4456.319619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4456.337821] alg: No test for adler32 (adler32-zlib) [ 4457.334224] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 4457.591703] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4459.233259] Key type lgssc registered [ 4460.390507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4493.003497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4493.054473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.477311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4494.520553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4494.661503] Lustre: lustre-MDT0000: new disk, initializing [ 4494.742603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.756607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4498.999082] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4509.090986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.155142] Lustre: 148961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4509.191372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4509.194429] Lustre: Skipped 1 previous similar message [ 4509.241621] Lustre: lustre-MDT0001: new disk, initializing [ 4509.293337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4509.311246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4509.316376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4514.419904] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4524.758123] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4531.222610] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4537.918068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4538.322912] Lustre: lustre-OST0000: new disk, initializing [ 4538.328954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4538.359685] Lustre: 150931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4538.495043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4544.107950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4544.124240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4544.238925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4545.716783] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4555.993938] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4561.455991] Lustre: Failing over lustre-OST0000 [ 4561.569963] Lustre: server umount lustre-OST0000 complete [ 4579.680180] Lustre: 147189:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449250/real 1789449250] req@00000000b35b975b x1876373425653888/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789449266 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4579.688977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.333893] Lustre: 147186:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449252/real 1789449252] req@00000000d234ce3f x1876373425655168/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789449268 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4581.385986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.181505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4582.432387] Lustre: 147188:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449252/real 1789449252] req@00000000ddec60cc x1876373425654912/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789449268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4582.468520] Lustre: 147188:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4582.660645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4582.686972] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4585.761959] Lustre: 147187:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449257/real 1789449257] req@000000004e4006a6 x1876373425656192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789449273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4585.779477] Lustre: 147187:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4588.771525] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4589.640059] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4590.816506] Lustre: 147188:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449262/real 1789449262] req@00000000f3c8f307 x1876373425657472/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789449278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4590.871565] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4590.877813] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4594.172655] Lustre: Failing over lustre-MDT0000 [ 4594.579638] Lustre: server umount lustre-MDT0000 complete [ 4596.205847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4596.211221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.277570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.310141] Lustre: Skipped 1 previous similar message [ 4600.522177] LustreError: 148969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4600.539740] LustreError: 148969:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4604.389309] LustreError: 148974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.404088] LustreError: 148974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4605.634827] LustreError: 150880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.506772] LustreError: 148975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.548191] LustreError: 148975:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4613.903317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4613.998234] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4614.252594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.307312] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4615.873321] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4619.254536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4619.262588] Lustre: Skipped 1 previous similar message [ 4619.282329] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4619.347637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4619.526537] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4629.479761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4629.482936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4629.489745] Lustre: Skipped 1 previous similar message [ 4629.505782] Lustre: Skipped 1 previous similar message [ 4633.906451] Lustre: server umount lustre-OST0000 complete [ 4639.715972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4639.723367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.730717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4639.759645] Lustre: Skipped 1 previous similar message [ 4644.181266] Lustre: server umount lustre-MDT0000 complete [ 4644.833436] LustreError: 151853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4644.853021] LustreError: 151853:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4647.749546] LustreError: 148956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449335 with bad export cookie 11048293101676634157 [ 4647.751495] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4647.763313] LustreError: 148956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4648.167534] Lustre: server umount lustre-MDT0001 complete [ 4659.682779] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4662.799301] Key type lgssc unregistered [ 4663.108802] LNet: 154582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4663.114910] LNetError: 154582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4663.129595] LNet: Removed LNI 192.168.202.134@tcp [ 4663.920268] Key type .llcrypt unregistered [ 4663.921600] Key type ._llcrypt unregistered [ 4700.907300] Key type ._llcrypt registered [ 4700.908431] Key type .llcrypt registered [ 4701.089381] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4702.226705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4702.394104] alg: No test for adler32 (adler32-zlib) [ 4703.573876] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 4703.748739] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4705.384220] Key type lgssc registered [ 4706.558168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4714.820535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4714.834181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.640389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.278092] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4725.742388] LustreError: 156101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4725.768681] LustreError: 156101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4730.849452] LustreError: 156102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4731.305936] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4735.753534] LustreError: 156130:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4735.762623] LustreError: 156130:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4735.768147] LustreError: 156130:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4736.063380] Lustre: server umount lustre-MDT0000 complete [ 4753.768492] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4762.638802] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4801.068027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.371923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4801.394986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4801.500641] Lustre: lustre-MDT0000: new disk, initializing [ 4801.642149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.661213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4807.225299] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4817.519980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4817.681231] Lustre: 159435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4817.766205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4817.768617] Lustre: Skipped 1 previous similar message [ 4817.855729] Lustre: lustre-MDT0001: new disk, initializing [ 4817.948516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4817.991306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4818.017747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4824.408764] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4837.137576] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4843.556728] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4851.786468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4852.339726] Lustre: lustre-OST0000: new disk, initializing [ 4852.358190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4852.378250] Lustre: 161404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4852.522616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4859.633113] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 4862.114635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4862.139150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4862.244672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4870.770508] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4875.549547] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4878.344136] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:19:23 (1789449563) [ 5150.691774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5150.697174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.720123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5154.281113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.281395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5154.303745] Lustre: Skipped 1 previous similar message [ 5154.314789] Lustre: Skipped 1 previous similar message [ 5155.743725] Lustre: server umount lustre-MDT0000 complete [ 5159.393166] LustreError: 159448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.407916] LustreError: 159448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5160.528375] LustreError: 159427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449847 with bad export cookie 13252267371344013731 [ 5160.536475] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5160.551718] LustreError: 159427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5164.518380] LustreError: 159444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.522396] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.582447] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5164.592492] Lustre: Skipped 1 previous similar message [ 5166.879568] Lustre: server umount lustre-MDT0001 complete [ 5178.142402] Lustre: server umount lustre-OST0000 complete [ 5188.528728] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 5200.106180] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5248.580154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5248.872231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5248.935685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5249.056668] Lustre: lustre-MDT0000: new disk, initializing [ 5249.245631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.285768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5257.708307] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5273.552257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.816620] Lustre: 166074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5273.939673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5273.947652] Lustre: Skipped 1 previous similar message [ 5274.093691] Lustre: lustre-MDT0001: new disk, initializing [ 5274.271198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5274.394421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5274.420844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5282.008833] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5294.530690] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5302.745983] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5312.018676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5312.417568] Lustre: lustre-OST0000: new disk, initializing [ 5312.424472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5312.433057] Lustre: 168044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5312.572256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5318.744492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5318.750522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5318.846494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5320.030363] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5332.196855] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5337.706552] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5337.982659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5343.479926] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5343.859794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5349.857923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5349.880489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.887705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5349.911651] Lustre: Skipped 1 previous similar message [ 5349.938114] Lustre: Skipped 2 previous similar messages [ 5352.334942] Lustre: server umount lustre-OST0000 complete [ 5360.101975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5360.125773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5363.452294] Lustre: server umount lustre-MDT0000 complete [ 5365.217573] LustreError: 167080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.248495] LustreError: 167080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5367.867909] LustreError: 166067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450055 with bad export cookie 13252267371344172561 [ 5367.875358] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5367.877110] LustreError: 166067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5368.190382] Lustre: server umount lustre-MDT0001 complete [ 5378.619815] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:27:44 (1789450064) [ 5398.278975] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 5409.871239] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5459.662628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.023196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5460.060716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5460.182208] Lustre: lustre-MDT0000: new disk, initializing [ 5460.325756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5460.372420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5464.656212] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5478.159937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5478.234841] Lustre: 172808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5478.457710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5483.504901] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5495.466226] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5503.074459] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5512.039202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5512.471149] Lustre: lustre-OST0000: new disk, initializing [ 5512.473766] Lustre: Skipped 1 previous similar message [ 5512.483390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5512.487885] Lustre: Skipped 2 previous similar messages [ 5512.507569] Lustre: 174789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5519.771489] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5521.504892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5521.512662] Lustre: Skipped 1 previous similar message [ 5521.518306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5521.608067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5531.445852] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5542.372743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.374265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5542.412384] Lustre: Skipped 1 previous similar message [ 5542.417801] LustreError: Skipped 1 previous similar message [ 5542.423986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5542.423996] Lustre: Skipped 1 previous similar message [ 5545.440372] Lustre: server umount lustre-OST0000 complete [ 5550.053963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5556.162658] Lustre: server umount lustre-MDT0000 complete [ 5557.731622] LustreError: 172814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5557.773736] LustreError: 172814:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5562.065692] LustreError: 172798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450249 with bad export cookie 13252267371344173443 [ 5562.071441] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5562.086726] LustreError: 172798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5563.030866] Lustre: server umount lustre-MDT0001 complete [ 5574.345590] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:30:59 (1789450259) [ 5591.905147] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 5604.656559] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5649.518996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5649.852223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5649.887281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5650.011850] Lustre: lustre-MDT0000: new disk, initializing [ 5650.116568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.119969] Lustre: Skipped 2 previous similar messages [ 5650.138510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5656.260196] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5668.661905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.815988] Lustre: 179112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5669.319449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5674.846742] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5686.234463] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5693.431685] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5701.688804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5702.163233] Lustre: 181087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5709.768930] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5709.950084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5710.219187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5721.961734] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5735.908040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5735.909282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.920345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5735.920353] Lustre: Skipped 4 previous similar messages [ 5735.951546] Lustre: Skipped 4 previous similar messages [ 5737.578310] Lustre: server umount lustre-OST0000 complete [ 5746.152787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5746.182727] LustreError: Skipped 1 previous similar message [ 5749.252103] Lustre: server umount lustre-MDT0000 complete [ 5751.268278] LustreError: 179123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.309924] LustreError: 179123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5754.796654] LustreError: 179105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450442 with bad export cookie 13252267371344174528 [ 5754.821392] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5755.344696] Lustre: server umount lustre-MDT0001 complete [ 5768.971524] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5772.140336] Key type lgssc unregistered [ 5772.711201] LNet: 182942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5772.735323] LNetError: 182942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5772.763340] LNet: Removed LNI 192.168.202.134@tcp [ 5773.932466] Key type .llcrypt unregistered [ 5773.934593] Key type ._llcrypt unregistered [ 5807.023890] Key type ._llcrypt registered [ 5807.033271] Key type .llcrypt registered [ 5807.288086] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5808.673197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5808.688776] alg: No test for adler32 (adler32-zlib) [ 5810.337699] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 5810.867993] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5812.656224] Key type lgssc registered [ 5815.078460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5825.060514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5825.089097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.866901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.410428] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5837.282226] LustreError: 184359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.295374] LustreError: 184359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5839.063290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.274275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5843.795963] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5853.418178] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5861.745418] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5871.531260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5871.967992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5873.021582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5883.533326] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 5895.475486] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5919.203444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5919.210609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.259718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5920.232605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5920.233093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5920.264495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5922.035534] Lustre: server umount lustre-OST0000 complete [ 5930.467292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.469369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5930.475432] Lustre: Skipped 1 previous similar message [ 5930.486243] Lustre: Skipped 1 previous similar message [ 5933.736291] Lustre: server umount lustre-MDT0000 complete [ 5935.601233] LustreError: 184359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5935.655196] LustreError: 184359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5938.589403] LustreError: 184340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450625 with bad export cookie 6038599243950594387 [ 5938.593746] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5938.610394] LustreError: 184340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5939.050609] Lustre: server umount lustre-MDT0001 complete [ 5953.390593] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 5958.398335] Key type lgssc unregistered [ 5958.949432] LNet: 187916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5958.959450] LNetError: 187916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5958.987402] LNet: Removed LNI 192.168.202.134@tcp [ 5959.933790] Key type .llcrypt unregistered [ 5959.936417] Key type ._llcrypt unregistered [ 5986.207304] Key type ._llcrypt registered [ 5986.211209] Key type .llcrypt registered [ 5986.401381] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 5988.134218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5988.213509] alg: No test for adler32 (adler32-zlib) [ 5989.383800] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 5989.743337] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 5991.496246] Key type lgssc registered [ 5992.850810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6001.916042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6001.941584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6003.742642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6009.612749] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6013.922634] LustreError: 189309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.951653] LustreError: 189309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6019.056088] LustreError: 189310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6020.793930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6021.084468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6026.458608] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6036.418618] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6043.481710] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6052.472502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6053.173236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.474239] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6072.673383] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6074.756684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6083.814173] Lustre: lustre-OST0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 6085.109997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6085.111595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6085.140821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6088.913886] Lustre: lustre-OST0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 6088.923687] Lustre: Skipped 1 previous similar message [ 6094.040856] Lustre: lustre-OST0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 6094.055694] Lustre: Skipped 2 previous similar messages [ 6097.376185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6097.722992] Lustre: server umount lustre-OST0000 complete [ 6109.899834] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9fa9042ee000.ost_server_uuid 50 [ 6112.386707] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6130.146498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6130.175338] Lustre: Skipped 1 previous similar message [ 6130.191888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6130.197182] Lustre: Skipped 2 previous similar messages [ 6131.836527] Lustre: server umount lustre-MDT0000 complete [ 6135.268930] LustreError: 189310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.306791] LustreError: 189310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6136.580121] LustreError: 189289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450823 with bad export cookie 3213216800645620943 [ 6136.584843] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6136.599814] LustreError: 189289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6137.464686] Lustre: server umount lustre-MDT0001 complete [ 6148.741890] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:40:34 (1789450834) [ 6155.975565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.563672] LustreError: 193231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6156.607263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6161.762531] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6161.890115] LustreError: 193232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.111214] LustreError: 193231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.146189] LustreError: 193231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6171.573790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.054546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6176.526587] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6185.842476] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6193.751914] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6204.064717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6204.305846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6212.461089] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6222.461331] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6225.928448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6231.303548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6231.627447] Lustre: lustre-OST0001: new disk, initializing [ 6231.631413] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6231.637911] Lustre: 196075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6231.717506] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6238.929892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6238.945052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6239.129712] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6241.537843] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6253.093505] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6256.609531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6256.615452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.636241] Lustre: Skipped 1 previous similar message [ 6256.648180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6258.659285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6258.665113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.694136] Lustre: server umount lustre-OST0000 complete [ 6264.538967] LustreError: 196541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.555201] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6270.851562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6276.956638] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6277.188561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6294.627645] Lustre: server umount lustre-OST0001 complete [ 6300.129852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6300.137757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.147265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6300.156300] Lustre: Skipped 4 previous similar messages [ 6309.044923] Lustre: server umount lustre-MDT0000 complete [ 6310.880327] Lustre: 188620:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789450982/real 1789450982] req@00000000afdf9786 x1876375033193728/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789450998 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6310.899200] LustreError: 194820:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.924082] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.924091] Lustre: Skipped 1 previous similar message [ 6310.982533] LustreError: 194820:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 6313.248271] LustreError: 193211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451000 with bad export cookie 3213216800645622014 [ 6313.257146] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6313.262521] LustreError: 193211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6313.696806] Lustre: server umount lustre-MDT0001 complete [ 6344.734917] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:43:49 (1789451029) [ 6354.635835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6354.973445] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6355.000635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6355.154137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.896352] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6372.129284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.363455] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6372.425453] Lustre: 199516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6378.272297] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6388.525442] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6396.060976] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6404.783374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6405.079423] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6412.415264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6414.046139] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6423.910848] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6431.967519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6432.035782] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6432.275721] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6432.278750] Lustre: Skipped 2 previous similar messages [ 6438.115154] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6440.019617] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6449.734720] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6453.740598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6453.759456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.786734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6453.789518] Lustre: Skipped 3 previous similar messages [ 6460.032405] Lustre: server umount lustre-OST0000 complete [ 6461.145540] LustreError: 201368:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.329871] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6479.342017] LustreError: Skipped 2 previous similar messages [ 6480.804366] Lustre: server umount lustre-OST0001 complete [ 6489.571123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.589702] Lustre: Skipped 4 previous similar messages [ 6493.028973] Lustre: server umount lustre-MDT0000 complete [ 6497.852322] LustreError: 199509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451185 with bad export cookie 3213216800645623414 [ 6497.859156] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6498.045690] Lustre: server umount lustre-MDT0001 complete [ 6528.778920] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:46:54 (1789451214) [ 6547.356467] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 6567.931948] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 6618.520777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6618.859816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6618.887525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6618.986403] Lustre: lustre-MDT0000: new disk, initializing [ 6619.104974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6619.131499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6624.612713] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6637.260372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6637.491643] Lustre: 207640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6637.576509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6637.582968] Lustre: Skipped 1 previous similar message [ 6637.719342] Lustre: lustre-MDT0001: new disk, initializing [ 6637.870910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6637.889992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6643.945960] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6656.750643] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6664.822575] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6673.629285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6674.131274] Lustre: lustre-OST0000: new disk, initializing [ 6674.135281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6674.150911] Lustre: 209611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6679.671774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6679.677980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6679.795665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6683.223119] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6695.271776] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6701.656336] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6701.907643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6707.521105] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6707.767940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6710.759357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6710.762135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6710.762505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6710.779393] Lustre: Skipped 1 previous similar message [ 6710.814804] Lustre: Skipped 8 previous similar messages [ 6724.066054] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6724.284370] Lustre: server umount lustre-OST0000 complete [ 6735.408579] Lustre: server umount lustre-MDT0000 complete [ 6736.361293] LustreError: 207651:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6736.387425] LustreError: 207651:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6739.901606] LustreError: 207631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451427 with bad export cookie 3213216800645625206 [ 6739.906133] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.923414] LustreError: 207631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6740.334870] Lustre: server umount lustre-MDT0001 complete [ 6750.876093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.083643] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6767.289607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6773.698586] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6783.025809] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6789.600489] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6796.923147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.683191] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6815.590433] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6821.012515] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6821.191602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6826.413548] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6826.760627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6831.073088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6831.089943] LustreError: Skipped 2 previous similar messages [ 6841.335675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6841.344043] Lustre: Skipped 11 previous similar messages [ 6843.360849] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6843.472076] Lustre: server umount lustre-OST0000 complete [ 6851.556593] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6858.241139] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6889.605624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6890.549565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6890.563974] Lustre: Skipped 5 previous similar messages [ 6896.625344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6896.663563] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6896.674350] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6896.737515] Lustre: Skipped 1 previous similar message [ 6901.770174] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6910.962350] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6912.769736] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6922.216294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6922.219762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6922.227320] Lustre: Skipped 5 previous similar messages [ 6925.014857] Lustre: server umount lustre-OST0000 complete [ 6935.675366] Lustre: server umount lustre-MDT0000 complete [ 6939.637703] LustreError: 211813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451626 with bad export cookie 3213216800645626088 [ 6939.641157] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6939.645387] LustreError: 211813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6939.902581] Lustre: server umount lustre-MDT0001 complete [ 6950.074721] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:53:55 (1789451635) [ 6959.468841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.922895] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6981.581228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6989.496395] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 6999.954297] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7007.287982] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7015.200641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7023.300460] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7031.512172] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7037.062156] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7037.311717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7042.260258] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7042.523706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7049.574828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7049.811232] Lustre: lustre-OST0001: new disk, initializing [ 7049.815352] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7049.823623] Lustre: 221180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7056.819177] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7058.002368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7058.013342] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7058.192483] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7066.653613] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7071.711789] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7071.989819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7077.295426] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7077.587196] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7084.022166] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7084.031764] LustreError: Skipped 2 previous similar messages [ 7094.241323] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7094.330908] Lustre: server umount lustre-OST0001 complete [ 7099.406607] LustreError: 219629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.443595] LustreError: 219629:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 7102.532956] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7102.911647] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7108.270987] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7108.518225] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7138.437271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7143.935530] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7143.973394] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7143.973706] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7144.035567] Lustre: Skipped 1 previous similar message [ 7145.635442] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7155.822989] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7157.722982] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7164.389991] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7164.397661] Lustre: Skipped 12 previous similar messages [ 7166.526539] Lustre: server umount lustre-OST0001 complete [ 7188.973291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.994228] Lustre: Skipped 6 previous similar messages [ 7200.228666] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7200.487926] Lustre: server umount lustre-OST0000 complete [ 7211.876031] Lustre: server umount lustre-MDT0000 complete [ 7216.324193] LustreError: 221178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451903 with bad export cookie 3213216800645627054 [ 7216.325362] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7216.343489] LustreError: 221178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7222.843386] Lustre: server umount lustre-MDT0001 complete [ 7253.386878] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:58:58 (1789451938) [ 7262.057529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.535358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7262.605429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7268.312306] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7278.106533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7278.198779] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7278.220810] Lustre: 226431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7284.059689] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7293.687653] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7301.031975] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7308.940666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7309.287619] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7316.654923] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7326.251483] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7337.169242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7337.239205] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7345.176625] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7353.992970] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7356.000301] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7361.434453] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7361.646090] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7367.386336] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7367.616403] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7374.840705] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fa9058cc800.ost_server_uuid 50 [ 7376.721242] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fa9058cc800.ost_server_uuid in IDLE state after 0 sec [ 7377.818750] Lustre: Permanently deactivating lustre-OST0001 [ 7377.830655] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7381.819547] Lustre: Permanently reactivating lustre-OST0001 [ 7381.848596] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7381.856808] Lustre: Skipped 2 previous similar messages [ 7388.154051] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7388.170103] LustreError: Skipped 3 previous similar messages [ 7392.474523] Lustre: server umount lustre-OST0001 complete [ 7411.168302] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7411.338740] Lustre: server umount lustre-OST0000 complete [ 7422.003258] Lustre: server umount lustre-MDT0000 complete [ 7426.348302] LustreError: 228294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452113 with bad export cookie 3213216800645628342 [ 7426.357798] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7426.363928] LustreError: 228294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7426.983917] Lustre: server umount lustre-MDT0001 complete [ 7455.993456] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:02:22 (1789452142) [ 7464.559292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.028388] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7465.070812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7465.086250] Lustre: Skipped 2 previous similar messages [ 7465.358635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.368488] Lustre: Skipped 9 previous similar messages [ 7470.349921] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7480.335904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.432614] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7480.490417] Lustre: 233140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7486.110553] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7494.179624] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7501.211417] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7509.272850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7509.494635] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7509.567773] Lustre: Permanently deactivating lustre-OST0000 [ 7509.597099] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7515.208046] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7515.329299] LustreError: 235269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 7515.343024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7517.820891] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7526.982988] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7535.944836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7536.023103] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7543.507481] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7544.638028] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7553.180261] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7555.007958] Lustre: Permanently reactivating lustre-OST0000 [ 7555.026192] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7555.036195] Lustre: Skipped 2 previous similar messages [ 7560.765716] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7560.772099] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7560.786936] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7560.849343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7560.863815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7578.862536] Lustre: server umount lustre-OST0001 complete [ 7588.831699] Lustre: server umount lustre-OST0000 complete [ 7607.264660] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7607.716510] Lustre: server umount lustre-MDT0000 complete [ 7611.832529] LustreError: 235008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452299 with bad export cookie 3213216800645629980 [ 7611.835277] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7611.842294] LustreError: 235008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7612.276775] Lustre: server umount lustre-MDT0001 complete [ 7624.418382] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 7627.592769] Key type lgssc unregistered [ 7628.013791] LNet: 238031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7628.022163] LNetError: 238031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7628.060563] LNet: Removed LNI 192.168.202.134@tcp [ 7629.163546] Key type .llcrypt unregistered [ 7629.167151] Key type ._llcrypt unregistered [ 7654.356632] Key type ._llcrypt registered [ 7654.359714] Key type .llcrypt registered [ 7654.520851] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7656.128928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7656.182767] alg: No test for adler32 (adler32-zlib) [ 7657.331892] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 7657.552616] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 7659.208728] Key type lgssc registered [ 7660.602983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7674.060678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7674.124236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.763779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.332650] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7686.117736] LustreError: 239497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.148662] LustreError: 239497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7691.026692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7691.192177] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7691.221204] Lustre: 239483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7691.235923] Lustre: Permanently deactivating lustre-MDT0001 [ 7691.240970] LustreError: 239496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7691.258450] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7691.554054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7692.546402] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7692.558761] LustreError: 239527:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7692.590288] LustreError: 239527:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7692.597370] LustreError: 239527:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7697.064027] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7705.988527] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7711.403811] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7718.541115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7718.761402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7720.820066] LustreError: 241340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7720.857246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7725.368676] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7726.053562] LustreError: 241340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7731.177178] LustreError: 241341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7733.713935] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7736.306563] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7740.929406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7741.157787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7746.552338] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7746.566195] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7747.804886] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7755.428328] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7759.267817] Lustre: Permanently reactivating lustre-MDT0001 [ 7759.285960] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7759.292309] Lustre: Skipped 2 previous similar messages [ 7765.213690] Lustre: lustre-MDT0001: Client dce23a12-874f-435e-825f-fda11dc83ad8 (at 192.168.202.34@tcp) reconnecting [ 7771.878520] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7773.694652] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7780.146599] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7780.459156] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7785.589483] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7785.825897] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7788.591291] Lustre: Permanently deactivating lustre-MDT0001 [ 7788.600573] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7788.603326] Lustre: Skipped 2 previous similar messages [ 7797.411587] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7803.432762] Lustre: server umount lustre-MDT0000 complete [ 7804.417342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7804.428191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.461625] LustreError: 239497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.486635] LustreError: 239497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7808.445132] LustreError: 239478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452495 with bad export cookie 7762237400062121957 [ 7808.451714] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7808.452493] LustreError: 239478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7809.083063] Lustre: server umount lustre-MDT0001 complete [ 7814.145981] Lustre: server umount lustre-OST0000 complete [ 7827.354641] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:08:33 (1789452513) [ 7840.663701] Lustre: server umount lustre-OST0001 complete [ 7848.754607] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 7859.781259] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 7902.528183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.858861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7902.875244] Lustre: Skipped 2 previous similar messages [ 7902.918188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7903.064389] Lustre: lustre-MDT0000: new disk, initializing [ 7903.152819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.180887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7908.443537] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7919.546408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7919.723061] Lustre: 247754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7919.835200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7919.840686] Lustre: Skipped 1 previous similar message [ 7919.954993] Lustre: lustre-MDT0001: new disk, initializing [ 7920.042314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7920.081499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7920.088793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7924.444121] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7936.336460] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7943.209553] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7949.981840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7950.458892] Lustre: lustre-OST0000: new disk, initializing [ 7950.471442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7950.490684] Lustre: 249725:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7950.697689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.638615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7957.654636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7957.817396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7958.494297] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7968.512271] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7970.436964] LustreError: 247763:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7979.969113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7980.144845] Lustre: lustre-OST0001: new disk, initializing [ 7980.153157] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7980.159645] Lustre: 250842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7980.460601] LustreError: 247763:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7980.495454] LustreError: 247762:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7986.771717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7986.782552] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7986.966124] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7988.153350] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 7990.536807] LustreError: 247762:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8000.012152] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8001.905822] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8007.649406] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8007.650121] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8007.664495] LustreError: Skipped 1 previous similar message [ 8007.669221] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8007.681028] Lustre: Skipped 4 previous similar messages [ 8007.708216] Lustre: Skipped 1 previous similar message [ 8012.772744] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8012.801610] Lustre: Skipped 1 previous similar message [ 8017.894227] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8017.903361] Lustre: Skipped 1 previous similar message [ 8018.401266] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8018.567196] Lustre: server umount lustre-OST0001 complete [ 8023.029723] LustreError: 249711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8023.054792] LustreError: 249711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8038.377150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8038.383050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8038.386926] Lustre: Skipped 1 previous similar message [ 8038.401086] Lustre: Skipped 1 previous similar message [ 8042.528404] Lustre: server umount lustre-OST0000 complete [ 8048.098135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8048.106400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.125814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8053.071453] Lustre: server umount lustre-MDT0000 complete [ 8057.112364] LustreError: 247748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452744 with bad export cookie 7762237400062124764 [ 8057.119310] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8057.124560] LustreError: 247748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8063.683205] Lustre: server umount lustre-MDT0001 complete [ 8074.690762] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 8077.483505] Key type lgssc unregistered [ 8077.826956] LNet: 252876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8077.834577] LNetError: 252876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8077.849717] LNet: Removed LNI 192.168.202.134@tcp [ 8078.786313] Key type .llcrypt unregistered [ 8078.788539] Key type ._llcrypt unregistered [ 8121.922717] Key type ._llcrypt registered [ 8121.925316] Key type .llcrypt registered [ 8122.100724] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 8123.200918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8123.312531] alg: No test for adler32 (adler32-zlib) [ 8124.492511] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 8124.804609] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 8126.512900] Key type lgssc registered [ 8127.626840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8135.832071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8135.857509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.299402] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8137.354597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8137.529251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8141.622573] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8150.779104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8150.842927] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8150.864491] Lustre: 255080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8151.077654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8156.277863] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8163.962984] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8169.492699] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8176.561524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8176.873876] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8177.125546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8183.607138] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8184.981807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8190.044837] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8241.122952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8241.135897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8241.152057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8241.644321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8241.647031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8241.671598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8246.028811] Lustre: server umount lustre-OST0000 complete [ 8251.176423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8265.837354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8266.380535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8273.182135] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8281.085594] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8281.770754] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8281.802113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8281.806441] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8281.826121] Lustre: Skipped 1 previous similar message [ 8323.043054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8323.044045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.054912] LustreError: Skipped 1 previous similar message [ 8323.069340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.095050] Lustre: Skipped 1 previous similar message [ 8327.553400] Lustre: server umount lustre-OST0000 complete [ 8333.285211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8333.291434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8333.305417] Lustre: Skipped 2 previous similar messages [ 8333.326185] Lustre: Skipped 1 previous similar message [ 8338.391821] Lustre: server umount lustre-MDT0000 complete [ 8342.765501] LustreError: 255071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453030 with bad export cookie 7754283769636570536 [ 8342.775300] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8342.776445] LustreError: 255071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8343.223776] Lustre: server umount lustre-MDT0001 complete [ 8354.381198] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 8357.292236] Key type lgssc unregistered [ 8357.598930] LNet: 260501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8357.609975] LNetError: 260501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8358.631345] LNet: Removed LNI 192.168.202.134@tcp [ 8359.772877] Key type .llcrypt unregistered [ 8359.775785] Key type ._llcrypt unregistered [ 8385.727704] Key type ._llcrypt registered [ 8385.731275] Key type .llcrypt registered [ 8385.894538] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 8387.092949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8387.264725] alg: No test for adler32 (adler32-zlib) [ 8388.338929] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 8388.623626] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 8390.376278] Key type lgssc registered [ 8391.645395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8400.052474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8400.066920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.409707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8406.399103] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8411.622465] LustreError: 261918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8411.632203] LustreError: 261918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8414.585675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.956303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8420.390606] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8428.111943] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8434.168376] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8441.173784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8441.498867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8446.650178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8448.066742] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8455.766174] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8482.795894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8482.809442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.825433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8486.884326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8486.891135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8486.897935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8486.938317] Lustre: Skipped 1 previous similar message [ 8488.573364] Lustre: server umount lustre-OST0000 complete [ 8492.001650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.012948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8492.022742] Lustre: Skipped 1 previous similar message [ 8497.128767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8497.132929] Lustre: Skipped 2 previous similar messages [ 8502.244170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8502.253445] Lustre: Skipped 1 previous similar message [ 8506.336629] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8506.640309] Lustre: server umount lustre-MDT0000 complete [ 8507.368587] LustreError: 261918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8507.406501] LustreError: 261918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8510.981840] LustreError: 261900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453198 with bad export cookie 13737623597211962801 [ 8510.984325] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8510.994662] LustreError: 261900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8511.333817] Lustre: server umount lustre-MDT0001 complete [ 8521.859066] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 8524.380779] Key type lgssc unregistered [ 8524.796770] LNet: 266226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8524.809714] LNetError: 266226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8524.832991] LNet: Removed LNI 192.168.202.134@tcp [ 8525.625591] Key type .llcrypt unregistered [ 8525.627738] Key type ._llcrypt unregistered [ 8541.960335] Key type ._llcrypt registered [ 8541.964385] Key type .llcrypt registered [ 8542.061363] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 8543.403745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8543.445873] alg: No test for adler32 (adler32-zlib) [ 8544.542767] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 8544.938655] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 8546.696246] Key type lgssc registered [ 8548.127399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8555.917420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8555.937285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8557.379401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8562.278853] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8566.759888] LustreError: 267443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8566.772348] LustreError: 267443:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8570.426449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8570.769937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8576.311119] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8584.660315] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8590.886338] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8600.304954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8600.632145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8602.686068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8608.843704] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8616.230515] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8629.218906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.227876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.231017] Lustre: Skipped 1 previous similar message [ 8629.243384] Lustre: Skipped 1 previous similar message [ 8630.827579] Lustre: server umount lustre-OST0000 complete [ 8637.419320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8637.434213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8637.447886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.459764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.460777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8641.489927] Lustre: server umount lustre-MDT0000 complete [ 8644.581734] LustreError: 267443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8644.620305] LustreError: 267443:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8645.015166] LustreError: 267423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453332 with bad export cookie 14963758565411487413 [ 8645.021019] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8645.028399] LustreError: 267423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8645.333795] Lustre: server umount lustre-MDT0001 complete [ 8656.699656] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 8659.791884] Key type lgssc unregistered [ 8660.166636] LNet: 271121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8660.179742] LNetError: 271121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8660.201798] LNet: Removed LNI 192.168.202.134@tcp [ 8661.132843] Key type .llcrypt unregistered [ 8661.134599] Key type ._llcrypt unregistered [ 8687.158583] Key type ._llcrypt registered [ 8687.163232] Key type .llcrypt registered [ 8687.306550] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 8688.251394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8688.298147] alg: No test for adler32 (adler32-zlib) [ 8689.392652] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 8689.617227] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 8691.273201] Key type lgssc registered [ 8692.158501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8700.821214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8700.842762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.590845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.838425] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8712.682853] LustreError: 272535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8712.707057] LustreError: 272535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8717.831676] LustreError: 272535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8718.663460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.169846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8723.908908] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8733.923483] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8740.454167] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8747.974879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.328181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.678706] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8765.122863] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8771.957782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8800.744661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8800.765773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8800.785120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8802.788501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8802.800466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8802.817376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.851729] Lustre: Skipped 1 previous similar message [ 8804.676446] Lustre: server umount lustre-OST0000 complete [ 8811.493057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8811.499912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8811.525529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8818.145566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8818.160836] Lustre: Skipped 3 previous similar messages [ 8823.264169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8823.266147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8823.292537] Lustre: Skipped 1 previous similar message [ 8823.500755] Lustre: server umount lustre-MDT0000 complete [ 8827.291634] LustreError: 272515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453514 with bad export cookie 8883666639300761586 [ 8827.297811] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8827.304200] LustreError: 272515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8827.584549] Lustre: server umount lustre-MDT0001 complete [ 8838.780428] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 8841.270423] Key type lgssc unregistered [ 8841.516803] LNet: 276863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8841.521589] LNetError: 276863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8842.563791] LNet: Removed LNI 192.168.202.134@tcp [ 8843.547845] Key type .llcrypt unregistered [ 8843.549358] Key type ._llcrypt unregistered [ 8861.673237] Key type ._llcrypt registered [ 8861.676519] Key type .llcrypt registered [ 8861.818635] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 8862.751236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8862.786843] alg: No test for adler32 (adler32-zlib) [ 8863.811421] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 8864.058803] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 8865.800450] Key type lgssc registered [ 8866.843796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8873.894096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8873.906881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8875.496135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.425621] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8885.736069] LustreError: 278113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8885.768463] LustreError: 278113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8890.858748] LustreError: 278115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8891.436056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8891.856349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8898.108510] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8906.332755] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8913.147889] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8921.347878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8921.508758] LustreError: 279959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8921.834146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8927.221210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8928.859928] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 8937.063853] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8952.802893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8952.815123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.816857] LustreError: Skipped 1 previous similar message [ 8952.819192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8952.831124] Lustre: Skipped 1 previous similar message [ 8952.851996] Lustre: Skipped 1 previous similar message [ 8955.184201] Lustre: server umount lustre-OST0000 complete [ 8963.047282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8963.048760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8963.055954] Lustre: Skipped 1 previous similar message [ 8963.061986] Lustre: Skipped 1 previous similar message [ 8965.984885] Lustre: server umount lustre-MDT0000 complete [ 8968.162700] LustreError: 278114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8968.188029] LustreError: 278114:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8969.579413] LustreError: 278080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453656 with bad export cookie 10601259325277958022 [ 8969.583701] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8969.587884] LustreError: 278080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8969.958983] Lustre: server umount lustre-MDT0001 complete [ 8984.373913] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 8987.423847] Key type lgssc unregistered [ 8987.665236] LNet: 281814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8987.677368] LNetError: 281814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8987.701980] LNet: Removed LNI 192.168.202.134@tcp [ 8988.497872] Key type .llcrypt unregistered [ 8988.500384] Key type ._llcrypt unregistered [ 9001.434892] Key type ._llcrypt registered [ 9001.437559] Key type .llcrypt registered [ 9001.540626] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9017.664825] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 9033.634965] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 9035.051253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9035.110749] alg: No test for adler32 (adler32-zlib) [ 9036.711427] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 9036.984696] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 9038.657536] Key type lgssc registered [ 9040.347545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9078.646926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9078.692867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9080.117763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9080.165800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9080.283445] Lustre: lustre-MDT0000: new disk, initializing [ 9080.432636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.484525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9085.189703] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9098.290584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.539653] Lustre: 285366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9098.624156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9098.628601] Lustre: Skipped 1 previous similar message [ 9098.770898] Lustre: lustre-MDT0001: new disk, initializing [ 9098.993862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9099.071408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9099.083689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9104.801704] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9115.483629] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9121.261269] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9129.476941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9129.830186] Lustre: lustre-OST0000: new disk, initializing [ 9129.845084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9129.857876] Lustre: 287335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9130.061705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9136.746040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9136.752405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9136.921246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9138.098185] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9148.579977] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9154.103641] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9154.371846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9158.915146] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9159.187859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9162.736731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9162.748825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9162.763337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9162.781988] Lustre: Skipped 1 previous similar message [ 9167.798240] Lustre: server umount lustre-OST0000 complete [ 9172.966466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.969048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9172.989334] Lustre: Skipped 1 previous similar message [ 9173.000210] Lustre: Skipped 1 previous similar message [ 9178.087133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.103891] Lustre: Skipped 2 previous similar messages [ 9178.517099] Lustre: server umount lustre-MDT0000 complete [ 9183.079469] LustreError: 285357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453870 with bad export cookie 16262479057213497458 [ 9183.101397] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9183.425949] Lustre: server umount lustre-MDT0001 complete [ 9192.805587] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:31:18 (1789453878) [ 9203.575695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9204.306537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9209.314093] LustreError: 289747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9209.362486] LustreError: 289747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9209.655709] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9214.443419] LustreError: 289748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9219.560085] LustreError: 289747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9220.913133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9221.746244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9229.329192] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9239.677249] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9246.374740] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9254.704842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9255.068372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9261.494051] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9270.061995] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9287.139711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9287.142750] LustreError: Skipped 1 previous similar message [ 9287.149968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9287.165641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9290.210201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9291.485468] Lustre: server umount lustre-OST0000 complete [ 9296.280526] Lustre: server umount lustre-MDT0000 complete [ 9298.405371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9298.410573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9298.425480] LustreError: 290475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9300.477693] LustreError: 289728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453987 with bad export cookie 16262479057213498340 [ 9300.484942] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9300.490953] LustreError: 289728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9300.789660] Lustre: server umount lustre-MDT0001 complete [ 9312.816808] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 9316.726579] Key type lgssc unregistered [ 9317.075821] LNet: 293304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9317.087289] LNetError: 293304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9318.129680] LNet: Removed LNI 192.168.202.134@tcp [ 9319.373351] Key type .llcrypt unregistered [ 9319.376354] Key type ._llcrypt unregistered [ 9353.353203] Key type ._llcrypt registered [ 9353.362963] Key type .llcrypt registered [ 9353.518760] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 9368.364335] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 9369.553521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9369.652182] alg: No test for adler32 (adler32-zlib) [ 9370.790416] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 9371.072375] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 9372.888366] Key type lgssc registered [ 9374.189850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9434.630706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9434.670846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9435.983564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9436.033373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9436.128246] Lustre: lustre-MDT0000: new disk, initializing [ 9436.274507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9436.332216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9442.340713] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9456.461636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.583368] Lustre: 297133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9456.655429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9456.658781] Lustre: Skipped 1 previous similar message [ 9456.697484] Lustre: lustre-MDT0001: new disk, initializing [ 9456.764677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9456.794318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9456.804271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9462.752379] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9472.646802] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9479.384710] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9488.593363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9489.086080] Lustre: lustre-OST2710: new disk, initializing [ 9489.090556] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9489.113809] Lustre: 299103:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9489.289593] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9494.677377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9494.695302] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9494.898083] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9497.918511] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9508.592691] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9516.362193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9516.515615] Lustre: lustre-OST03e8: new disk, initializing [ 9516.518342] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9516.523930] Lustre: 300168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9516.639590] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9524.377796] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9526.372851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9526.383429] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9526.557862] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9533.086875] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9541.818728] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9542.356392] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9548.127195] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9548.459956] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9553.901483] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9554.127683] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9560.343424] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9560.663877] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9567.725195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.746304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9567.750576] Lustre: Skipped 3 previous similar messages [ 9567.761018] Lustre: Skipped 3 previous similar messages [ 9572.839416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9572.842599] Lustre: Skipped 3 previous similar messages [ 9575.524344] Lustre: server umount lustre-MDT0000 complete [ 9579.682555] LustreError: 297124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454266 with bad export cookie 12208429045930029434 [ 9579.684282] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9579.698703] LustreError: 297124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9580.065790] Lustre: server umount lustre-MDT0001 complete [ 9595.530459] Lustre: server umount lustre-OST2710 complete [ 9599.456243] Lustre: 294929:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789454270/real 1789454270] req@000000008a87c6cd x1876378578396672/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789454286 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9599.468805] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9600.171662] Lustre: server umount lustre-OST03e8 complete [ 9622.451603] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 9635.099220] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 9683.675956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9684.058844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9684.122199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9684.273110] Lustre: lustre-MDT0000: new disk, initializing [ 9684.491120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9684.509168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9690.685882] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9704.608995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9704.907559] Lustre: 305422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9704.969681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9704.979271] Lustre: Skipped 1 previous similar message [ 9705.026932] Lustre: lustre-MDT0001: new disk, initializing [ 9705.104111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9705.143180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9705.151117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9710.872221] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9722.395216] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9729.546103] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9738.600496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.867633] Lustre: lustre-OST0000: new disk, initializing [ 9738.871398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9738.879836] Lustre: 307393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9738.997564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9747.825548] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9748.095053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9748.110552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9748.367242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9759.932863] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9766.433918] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9766.623370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9774.400175] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9774.855457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9779.177492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9779.180030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.193281] LustreError: Skipped 1 previous similar message [ 9779.196026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9779.225463] Lustre: Skipped 1 previous similar message [ 9783.872809] Lustre: server umount lustre-OST0000 complete [ 9789.411272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9789.442723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9789.452701] Lustre: Skipped 2 previous similar messages [ 9794.532430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9794.545582] Lustre: Skipped 1 previous similar message [ 9795.645607] Lustre: server umount lustre-MDT0000 complete [ 9799.650075] LustreError: 305434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.674993] LustreError: 305434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9800.289598] LustreError: 306228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454487 with bad export cookie 12208429045930031128 [ 9800.292166] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9800.307752] LustreError: 306228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9800.688945] Lustre: server umount lustre-MDT0001 complete [ 9812.147429] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:41:38 (1789454498) [ 9814.011624] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9815.703399] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:41:41 (1789454501) [ 9820.085646] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 9852.101786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9852.423585] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9852.441131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9852.643871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9857.361897] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9866.996702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9867.075806] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9867.147853] Lustre: 311252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9873.184383] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 9897.080760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9897.381522] Lustre: Denying initial registration attempt from nid 192.168.202.134@tcp, specified as failover [ 9897.390740] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9897.397469] LustreError: 312708:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9897.402806] LustreError: 312708:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9897.406713] LustreError: 312708:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9897.460923] Lustre: server umount lustre-OST0000 complete [ 9897.464907] LustreError: 312708:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9904.611574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.614893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9904.632428] Lustre: Skipped 2 previous similar messages [ 9910.184210] Lustre: server umount lustre-MDT0000 complete [ 9913.810515] LustreError: 311244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454601 with bad export cookie 12208429045930032010 [ 9913.815224] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9913.818145] LustreError: 311244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9914.091600] Lustre: server umount lustre-MDT0001 complete [ 9927.514357] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 9931.189799] Key type lgssc unregistered [ 9931.514341] LNet: 313829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9931.520516] LNetError: 313829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9932.584633] LNet: Removed LNI 192.168.202.134@tcp [ 9934.200262] Key type .llcrypt unregistered [ 9934.201623] Key type ._llcrypt unregistered [ 9948.666605] Key type ._llcrypt registered [ 9948.670630] Key type .llcrypt registered [ 9948.793904] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 9950.453992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9950.586472] alg: No test for adler32 (adler32-zlib) [ 9951.745249] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [ 9952.093665] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 9953.792489] Key type lgssc registered [ 9955.414591] Lustre: Echo OBD driver; http://www.lustre.org/ [10005.022281] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [10014.880740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10014.899749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10016.204179] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10016.222722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10016.419343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10021.165727] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10030.632161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10030.712565] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10030.779626] Lustre: 316596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10031.090135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10035.955078] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10044.681934] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10050.299481] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10072.296760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10072.566789] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10072.770849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10078.504663] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10090.621677] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10094.052679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10094.073723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10094.082667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10094.091521] Lustre: Skipped 1 previous similar message [10094.099619] Lustre: Skipped 1 previous similar message [10099.174041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10099.187128] Lustre: Skipped 1 previous similar message [10099.906159] Lustre: server umount lustre-OST0000 complete [10107.891767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10107.906117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.943026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10109.410725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10114.529328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10114.537333] Lustre: Skipped 3 previous similar messages [10119.136170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10119.422734] Lustre: server umount lustre-MDT0000 complete [10123.681085] LustreError: 317346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454810 with bad export cookie 11658537091903618853 [10123.687129] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10123.695770] LustreError: 317346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10124.395681] Lustre: server umount lustre-MDT0001 complete [10138.231758] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [10142.270991] Key type lgssc unregistered [10142.710045] LNet: 320170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10142.721175] LNetError: 320170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10142.735756] LNet: Removed LNI 192.168.202.134@tcp [10143.760743] Key type .llcrypt unregistered [10143.766986] Key type ._llcrypt unregistered [10175.771553] Key type ._llcrypt registered [10175.775347] Key type .llcrypt registered [10175.909914] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [10177.154263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10177.220994] alg: No test for adler32 (adler32-zlib) [10178.683301] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [10179.004400] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [10180.832160] Key type lgssc registered [10182.163349] Lustre: Echo OBD driver; http://www.lustre.org/ [10191.255669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10191.307897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.040532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10198.085389] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10203.627034] LustreError: 321585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10203.643088] LustreError: 321585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10208.740964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10208.749063] LustreError: 321586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10209.232510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10214.774573] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10226.719327] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10234.713630] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10244.343963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10244.983940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10252.693664] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10263.761214] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10273.764449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10273.766201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10273.797490] Lustre: Skipped 1 previous similar message [10273.800433] Lustre: Skipped 2 previous similar messages [10275.562687] Lustre: server umount lustre-MDT0000 complete [10276.096506] LustreError: 321581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10278.890358] LustreError: 321586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10278.902386] LustreError: 321586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10280.219495] LustreError: 321565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454967 with bad export cookie 16160310886333866432 [10280.222346] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10280.229070] LustreError: 321565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10281.170453] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [10281.180879] Lustre: Skipped 1 previous similar message [10284.001968] LustreError: 321582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10284.004748] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10284.026280] LustreError: 321582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10284.066319] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10286.290717] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [10286.309198] Lustre: Skipped 1 previous similar message [10287.117920] Lustre: server umount lustre-MDT0001 complete [10295.339085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10305.674995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.961661] LustreError: 320894:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ea27e962 x1876379425612416/t0(0) o250->MGC192.168.202.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10325.465496] LustreError: 323419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.478139] LustreError: 323419:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10325.590518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10325.673931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10331.242561] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10339.820727] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10340.732299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.133953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10346.493626] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10346.774664] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10355.874079] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10357.434104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10364.522586] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10366.284844] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10376.673658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10376.687599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10376.712664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10379.007088] Lustre: server umount lustre-OST0000 complete [10387.428118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10387.437167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10387.442983] Lustre: Skipped 2 previous similar messages [10387.444171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10387.444176] Lustre: Skipped 2 previous similar messages [10389.580590] Lustre: server umount lustre-MDT0000 complete [10392.546487] LustreError: 325187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.569962] LustreError: 325187:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [10393.966429] LustreError: 324171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455081 with bad export cookie 16160310886333873971 [10393.969947] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10393.979825] LustreError: 324171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10394.464841] Lustre: server umount lustre-MDT0001 complete [10407.016418] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [10409.939239] Key type lgssc unregistered [10410.321631] LNet: 327857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10410.335737] LNetError: 327857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10411.385927] LNet: Removed LNI 192.168.202.134@tcp [10412.211730] Key type .llcrypt unregistered [10412.214782] Key type ._llcrypt unregistered [10436.867891] Key type ._llcrypt registered [10436.869641] Key type .llcrypt registered [10436.998934] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [10438.082376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10438.310687] alg: No test for adler32 (adler32-zlib) [10439.294462] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [10439.462413] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [10441.164683] Key type lgssc registered [10442.415044] Lustre: Echo OBD driver; http://www.lustre.org/ [10452.362596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10452.386965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10454.468655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10461.089321] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10463.720265] LustreError: 329271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10463.734275] LustreError: 329271:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10468.840166] LustreError: 329272:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.246828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.567460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10475.355578] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10484.210502] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10490.290512] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10496.240651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10496.247917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10496.261933] Lustre: Skipped 1 previous similar message [10499.601638] Lustre: server umount lustre-MDT0000 complete [10501.610794] LustreError: 330000:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10501.628426] LustreError: 330000:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10503.642831] LustreError: 329251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455190 with bad export cookie 6291737813303959358 [10503.659843] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10503.661908] LustreError: 329251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10504.012357] Lustre: server umount lustre-MDT0001 complete [10519.570593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10519.782654] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10519.801714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10519.940688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10525.611112] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10537.499867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10537.605790] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10537.666785] Lustre: 331862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10537.792067] LustreError: 331872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10538.043194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10542.746360] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10550.731277] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10558.637372] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10568.495733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10568.728920] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10568.747389] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10568.756248] LustreError: 333745:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10568.764080] LustreError: 333745:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10568.770676] LustreError: 333745:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10568.862532] Lustre: server umount lustre-OST0000 complete [10568.867431] LustreError: 333745:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10579.962245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10580.179586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10580.395213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10585.704258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10585.813963] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10593.999493] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10602.049940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10602.170877] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10602.249782] Lustre: lustre-OST0001: new disk, initializing [10602.253557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10602.267248] Lustre: 335288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10602.387413] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10608.249626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10608.265223] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10608.402106] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10609.445470] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10617.285230] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10623.972732] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10623.976790] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.980567] LustreError: Skipped 1 previous similar message [10623.988801] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10624.000502] Lustre: Skipped 1 previous similar message [10624.017242] Lustre: Skipped 2 previous similar messages [10629.103566] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10629.116019] Lustre: Skipped 1 previous similar message [10634.231119] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10634.237813] Lustre: Skipped 1 previous similar message [10635.232603] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10635.736033] Lustre: server umount lustre-OST0001 complete [10639.340530] LustreError: 334210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10639.381285] LustreError: 334210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10641.889601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10641.910289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.950921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10647.660827] Lustre: server umount lustre-OST0000 complete [10654.690660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10654.705666] Lustre: Skipped 1 previous similar message [10654.711394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10654.719950] Lustre: Skipped 4 previous similar messages [10659.210298] Lustre: server umount lustre-MDT0000 complete [10659.810714] LustreError: 331872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10659.839689] LustreError: 331872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10664.431908] LustreError: 331852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455351 with bad export cookie 6291737813303959918 [10664.449448] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10664.452156] LustreError: 331852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10665.072300] Lustre: server umount lustre-MDT0001 complete [10677.487422] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [10680.640859] Key type lgssc unregistered [10680.962172] LNet: 337245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10680.976480] LNetError: 337245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10681.002301] LNet: Removed LNI 192.168.202.134@tcp [10682.178375] Key type .llcrypt unregistered [10682.180071] Key type ._llcrypt unregistered [10761.626570] Key type ._llcrypt registered [10761.630434] Key type .llcrypt registered [10761.795834] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [10779.001514] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [10781.584251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10781.688735] alg: No test for adler32 (adler32-zlib) [10783.043290] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [10783.445699] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [10785.264220] Key type lgssc registered [10786.250175] Lustre: Echo OBD driver; http://www.lustre.org/ [10830.618809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10830.666176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10832.209400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10832.228830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10832.322658] Lustre: lustre-MDT0000: new disk, initializing [10832.473236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10832.498890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10837.414557] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10851.059216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.285993] Lustre: 342179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10851.387505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10851.403691] Lustre: Skipped 1 previous similar message [10851.552491] Lustre: lustre-MDT0001: new disk, initializing [10851.720797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10851.798316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10851.823578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10857.419440] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10870.102470] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10878.295161] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10887.293902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10887.757405] Lustre: lustre-OST0000: new disk, initializing [10887.763106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10887.773986] Lustre: 344149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10887.911622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10894.428651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10894.446158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10894.597237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10895.093220] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10907.610914] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10911.955531] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10912.306430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10917.477528] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10917.725194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10920.420873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10920.424811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.438263] LustreError: Skipped 1 previous similar message [10920.441262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10920.445303] Lustre: Skipped 1 previous similar message [10920.476068] Lustre: Skipped 1 previous similar message [10925.537594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10925.537594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10926.415304] Lustre: server umount lustre-OST0000 complete [10933.730357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10933.735759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10933.748366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10935.780773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10937.411086] Lustre: server umount lustre-MDT0000 complete [10942.599182] LustreError: 342174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455629 with bad export cookie 6408022957600827866 [10942.601574] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10942.614104] LustreError: 342174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10942.934389] Lustre: server umount lustre-MDT0001 complete [10953.961688] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:00:39 (1789455639) [10964.528732] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:00:50 (1789455650) [10976.477768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.955120] LustreError: 346917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.978795] LustreError: 346917:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10977.067398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10982.369402] LustreError: 346918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10982.398529] LustreError: 346918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10983.394957] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [10987.494534] LustreError: 346917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10992.610197] LustreError: 346918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10993.616737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10994.143147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11000.352393] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11009.785617] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11017.155273] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11025.597389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11026.096455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11034.791455] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11044.604115] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11059.030635] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11061.118460] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11063.122585] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11065.950381] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11068.356483] Lustre: Failing over lustre-MDT0000 [11068.390095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.423589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11068.435026] Lustre: Skipped 2 previous similar messages [11068.742419] Lustre: server umount lustre-MDT0000 complete [11072.271341] LustreError: 349282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11082.435569] LustreError: 349282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11082.454874] LustreError: 349282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [11088.641430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11088.710774] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11088.909573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11088.964779] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11092.682228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11094.002490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11094.048627] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11094.105609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11094.150804] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11104.714717] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11106.608816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11110.518482] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11112.793955] Lustre: Failing over lustre-MDT0000 [11113.142955] Lustre: server umount lustre-MDT0000 complete [11113.687626] LustreError: 348182:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11113.698730] LustreError: 348182:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11114.472356] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11114.502994] Lustre: Skipped 3 previous similar messages [11114.531092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11129.824327] Lustre: 340402:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789455801/real 1789455801] req@00000000576c42dd x1876380059009152/t0(0) o400->MGC192.168.202.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789455817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11129.846449] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11133.590671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11140.098870] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x58eddeedc458746f [11140.122481] Lustre: MGC192.168.202.134@tcp: Connection restored to 0@lo (at 0@lo) [11140.129951] Lustre: Skipped 2 previous similar messages [11140.296073] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (not set up) [11140.310246] Lustre: Skipped 2 previous similar messages [11140.579901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11140.687510] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11145.432878] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11145.726151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11145.816237] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11145.915052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11146.833445] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11158.601604] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11160.593059] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11171.299046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11171.309325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11171.319201] Lustre: Skipped 3 previous similar messages [11171.341403] Lustre: Skipped 2 previous similar messages [11173.760208] Lustre: server umount lustre-MDT0000 complete [11176.417946] LustreError: 347647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11176.440605] LustreError: 347647:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [11177.458125] LustreError: 346899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455864 with bad export cookie 6408022957600830575 [11177.461633] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11177.469256] LustreError: 346899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11177.941879] Lustre: server umount lustre-MDT0001 complete [11193.630856] Lustre: server umount lustre-OST0000 complete [11211.154500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11211.950921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11217.115793] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11226.562698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11232.769134] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11242.371346] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11249.575010] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11257.663991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11265.137497] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11274.295687] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11281.859865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11292.130068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11292.143310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11292.151539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11292.172191] Lustre: Skipped 1 previous similar message [11298.243699] Lustre: server umount lustre-OST0000 complete [11310.624426] Lustre: server umount lustre-MDT0000 complete [11312.612514] LustreError: 353292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11312.641086] LustreError: 353292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11316.399907] LustreError: 353272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456003 with bad export cookie 6408022957600831331 [11316.428811] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11316.451377] LustreError: 353272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11317.105463] Lustre: server umount lustre-MDT0001 complete [11330.629565] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [11333.901994] Key type lgssc unregistered [11334.257297] LNet: 356848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11334.264402] LNetError: 356848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11335.278733] LNet: Removed LNI 192.168.202.134@tcp [11336.670817] Key type .llcrypt unregistered [11336.672401] Key type ._llcrypt unregistered [11369.925981] Key type ._llcrypt registered [11369.929766] Key type .llcrypt registered [11370.092795] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [11371.838780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11371.885585] alg: No test for adler32 (adler32-zlib) [11373.274806] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [11373.596549] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [11375.424244] Key type lgssc registered [11376.613054] Lustre: Echo OBD driver; http://www.lustre.org/ [11385.256267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11385.271397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11386.790608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11392.233690] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11397.097602] LustreError: 358364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.135667] LustreError: 358364:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11401.959526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11402.210280] LustreError: 358363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11402.567396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11407.725368] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11418.988933] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11425.518968] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11434.142771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11434.851803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11443.844704] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11453.447350] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11457.440988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11472.878332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11472.884613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11472.889471] Lustre: Skipped 2 previous similar messages [11472.907743] Lustre: Skipped 2 previous similar messages [11476.911185] Lustre: server umount lustre-MDT0000 complete [11478.011938] LustreError: 358359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11478.048358] LustreError: 358359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11480.928225] LustreError: 358343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456168 with bad export cookie 6954825855866452706 [11480.931525] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11480.951415] LustreError: 358343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11481.505842] Lustre: server umount lustre-MDT0001 complete [11497.105593] Lustre: server umount lustre-OST0000 complete [11512.771481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11513.286565] LustreError: 362102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11513.399725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11518.433656] LustreError: 362103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11518.745199] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11527.902325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11528.237093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11533.656533] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11544.144989] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11551.446867] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11560.522789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11561.119655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11567.345956] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11576.319619] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11583.684942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11599.841280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11599.850895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11599.879422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11604.450672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.451208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11604.468409] Lustre: Skipped 2 previous similar messages [11605.855222] Lustre: server umount lustre-MDT0000 complete [11609.580146] LustreError: 362098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11609.591317] LustreError: 362098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11609.881831] LustreError: 362084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456297 with bad export cookie 6954825855866453819 [11609.906130] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11609.909236] LustreError: 362084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11610.324284] Lustre: server umount lustre-MDT0001 complete [11624.035608] Lustre: server umount lustre-OST0000 complete [11642.508205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11642.844780] LustreError: 366034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11642.930965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11647.766172] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11655.980733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11660.364415] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11670.609320] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11678.071636] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11686.447736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11686.769151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11686.775955] Lustre: Skipped 1 previous similar message [11687.811107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11693.601600] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11703.301388] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11711.460023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11711.464827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11711.477977] Lustre: Skipped 2 previous similar messages [11711.490216] Lustre: Skipped 1 previous similar message [11714.943841] Lustre: server umount lustre-OST0000 complete [11721.700515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.727914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11721.735058] Lustre: Skipped 1 previous similar message [11725.232532] Lustre: server umount lustre-MDT0000 complete [11726.819848] LustreError: 366035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11726.848244] LustreError: 366035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11729.313030] LustreError: 366017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456416 with bad export cookie 6954825855866454841 [11729.322417] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11729.341162] LustreError: 366017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11729.715990] Lustre: server umount lustre-MDT0001 complete [11742.422411] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [11745.053278] Key type lgssc unregistered [11745.315664] LNet: 369567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11745.323926] LNetError: 369567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11745.339288] LNet: Removed LNI 192.168.202.134@tcp [11746.193092] Key type .llcrypt unregistered [11746.194753] Key type ._llcrypt unregistered [11782.507246] Key type ._llcrypt registered [11782.510558] Key type .llcrypt registered [11782.648636] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [11784.272839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11784.601895] alg: No test for adler32 (adler32-zlib) [11785.831750] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [11786.200216] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [11787.976185] Key type lgssc registered [11789.189964] Lustre: Echo OBD driver; http://www.lustre.org/ [11798.732988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11798.751492] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.762398] LustreError: 371007:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11798.783878] LustreError: 371007:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11798.792267] LustreError: 371007:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11798.800635] LustreError: 371007:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11798.809322] LustreError: 371007:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11806.779721] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11806.783658] LustreError: 371405:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11806.796529] LustreError: 371405:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11806.804793] LustreError: 371405:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11806.808733] LustreError: 371405:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11806.814405] LustreError: 371405:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11829.080931] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [11832.115342] Key type lgssc unregistered [11832.450492] LNet: 372408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11832.456844] LNetError: 372408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11832.468121] LNet: Removed LNI 192.168.202.134@tcp [11833.398493] Key type .llcrypt unregistered [11833.402085] Key type ._llcrypt unregistered [11852.737555] Key type ._llcrypt registered [11852.740766] Key type .llcrypt registered [11852.854995] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [11869.319245] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [11870.890190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11871.046871] alg: No test for adler32 (adler32-zlib) [11872.246603] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [11872.599494] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [11874.376412] Key type lgssc registered [11875.796872] Lustre: Echo OBD driver; http://www.lustre.org/ [11922.254844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11922.341531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11923.939682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11924.036647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11924.210139] Lustre: lustre-MDT0000: new disk, initializing [11924.442357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11924.549773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11931.191330] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11944.046866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11944.335903] Lustre: 375566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11944.428576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11944.438847] Lustre: Skipped 1 previous similar message [11944.618215] Lustre: lustre-MDT0001: new disk, initializing [11944.856555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11944.933194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11944.954724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11950.962845] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11960.688971] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11966.734249] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11974.885728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11975.237738] Lustre: lustre-OST0000: new disk, initializing [11975.245635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11975.254785] Lustre: 377535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11975.388691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11980.886739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11980.904725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11981.011983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11982.903678] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [11993.447911] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11998.867311] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11999.112135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12004.282435] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12004.522919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12006.882251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12006.883191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12006.898189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12006.929786] Lustre: Skipped 1 previous similar message [12012.005470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12012.014460] Lustre: Skipped 1 previous similar message [12012.997950] Lustre: server umount lustre-OST0000 complete [12020.194366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12020.206182] Lustre: Skipped 2 previous similar messages [12020.207293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12020.224754] Lustre: Skipped 1 previous similar message [12024.120208] Lustre: server umount lustre-MDT0000 complete [12028.448979] LustreError: 375559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456715 with bad export cookie 12336882129312751468 [12028.458246] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12028.465691] LustreError: 375559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12028.955981] Lustre: server umount lustre-MDT0001 complete [12039.441492] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12041.470540] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:18:47 (1789456727) [12048.737620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12049.269156] LustreError: 379996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12049.294968] LustreError: 379996:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12049.398570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12054.430811] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12054.497504] LustreError: 379997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.617877] LustreError: 379996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12064.739765] LustreError: 379997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12065.377396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12065.936369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12071.798918] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12080.972760] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12087.620730] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12095.082546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12095.619611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12103.610112] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12113.181851] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12122.207776] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12122.430874] Lustre: lustre-OST0001: new disk, initializing [12122.433948] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12122.439268] Lustre: 382837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12122.530483] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12128.419572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12128.437445] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12128.569696] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12130.972684] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12143.672963] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12149.227372] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.239261] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12149.259292] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12149.269582] Lustre: Skipped 1 previous similar message [12149.294298] LustreError: Skipped 1 previous similar message [12154.299963] Lustre: server umount lustre-OST0001 complete [12154.364608] LustreError: 383087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12154.382237] LustreError: 383087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12164.595376] LustreError: 383087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12164.623906] LustreError: 383087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12174.822480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12174.836686] Lustre: Skipped 1 previous similar message [12174.843066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12174.847301] Lustre: Skipped 3 previous similar messages [12185.066865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12185.069029] LustreError: 381819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12185.076786] Lustre: Skipped 3 previous similar messages [12185.102277] LustreError: 381819:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [12188.128222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12188.309810] Lustre: server umount lustre-OST0000 complete [12193.773468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12193.784592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12199.158071] Lustre: server umount lustre-MDT0000 complete [12203.438163] LustreError: 379977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456890 with bad export cookie 12336882129312752343 [12203.443845] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12203.446588] LustreError: 379977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12210.315310] Lustre: server umount lustre-MDT0001 complete [12224.227797] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12227.293676] Key type lgssc unregistered [12227.655573] LNet: 384828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12227.666352] LNetError: 384828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12227.698715] LNet: Removed LNI 192.168.202.134@tcp [12228.878463] Key type .llcrypt unregistered [12228.881933] Key type ._llcrypt unregistered [12289.510692] Key type ._llcrypt registered [12289.513651] Key type .llcrypt registered [12289.660299] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [12291.190745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12291.348811] alg: No test for adler32 (adler32-zlib) [12292.488034] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [12292.702632] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [12294.368382] Key type lgssc registered [12295.369463] Lustre: Echo OBD driver; http://www.lustre.org/ [12302.598730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12302.632897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12303.964909] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12304.008683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12304.177926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12308.865994] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12317.421564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12317.574538] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12317.611233] Lustre: 387626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12317.914981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12322.613151] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12330.823538] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12337.792564] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12341.732085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12341.739950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12341.757740] Lustre: Skipped 1 previous similar message [12341.784875] Lustre: Skipped 1 previous similar message [12347.047443] Lustre: server umount lustre-MDT0000 complete [12351.975693] LustreError: 387634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12352.008823] LustreError: 387634:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12353.141026] LustreError: 387619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457040 with bad export cookie 7465860180529171449 [12353.143593] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12353.156828] LustreError: 387619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12353.738700] Lustre: server umount lustre-MDT0001 complete [12369.173536] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:24:14 (1789457054) [12379.046072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12380.148648] LustreError: 390142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12380.274251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12385.259347] LustreError: 390143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12387.462326] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12390.372400] LustreError: 390142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12395.490624] LustreError: 390143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.563286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12401.300138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12408.320584] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12418.095655] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12425.205302] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12433.164204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12433.475569] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12433.711805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12440.929557] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12450.054895] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12458.040767] Lustre: Permanently deactivating lustre-OST0000 [12458.072343] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12464.678715] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12465.808945] LustreError: 393081:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12465.830556] LustreError: 393081:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12465.846362] LustreError: 393081:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12470.079073] LustreError: 393131:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12470.096964] LustreError: 393131:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12478.673758] Lustre: server umount lustre-OST0000 complete [12482.297696] LustreError: 393381:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12482.304169] LustreError: 393381:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12486.629602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12486.666662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12486.678527] Lustre: Skipped 1 previous similar message [12491.745325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12491.751477] Lustre: Skipped 1 previous similar message [12496.893341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12496.916357] Lustre: Skipped 1 previous similar message [12498.402150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12498.821742] Lustre: server umount lustre-MDT0000 complete [12501.995407] LustreError: 390143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12502.015790] LustreError: 390143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12503.183947] LustreError: 390874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457190 with bad export cookie 7465860180529172030 [12503.186308] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12503.211107] LustreError: 390874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12512.900992] Lustre: server umount lustre-MDT0001 complete [12522.123769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12528.013873] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12531.017993] LustreError: 394541:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12537.527211] Lustre: 394880:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.134@tcp [12540.024787] Lustre: server umount MGS complete [12552.910803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12553.505407] LustreError: 395605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12553.548507] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12553.555589] Lustre: Skipped 1 previous similar message [12553.644430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12559.743578] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12570.101180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12570.377557] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12570.491156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12575.168674] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12583.334448] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12588.748485] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12590.958570] Lustre: Permanently reactivating lustre-OST0000 [12590.981598] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12590.999405] Lustre: Skipped 2 previous similar messages [12618.208503] LustreError: 397338:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12626.399465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12626.794627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12634.614857] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12644.003694] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12653.538823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12653.545968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.552637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12653.557665] LustreError: Skipped 1 previous similar message [12653.598100] Lustre: Skipped 2 previous similar messages [12657.979974] Lustre: server umount lustre-OST0000 complete [12663.779444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12663.787657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12663.808457] Lustre: Skipped 1 previous similar message [12663.814152] Lustre: Skipped 1 previous similar message [12669.660477] Lustre: server umount lustre-MDT0000 complete [12672.993990] LustreError: 395606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12673.020154] LustreError: 395606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12675.015928] LustreError: 395585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457362 with bad export cookie 7465860180529173080 [12675.016870] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12675.027389] LustreError: 395585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12675.486159] Lustre: server umount lustre-MDT0001 complete [12690.088189] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [12693.757321] Key type lgssc unregistered [12694.080790] LNet: 399923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12694.086825] LNetError: 399923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12694.097056] LNet: Removed LNI 192.168.202.134@tcp [12695.316814] Key type .llcrypt unregistered [12695.320830] Key type ._llcrypt unregistered [12719.330833] Key type ._llcrypt registered [12719.333840] Key type .llcrypt registered [12719.478123] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [12737.271844] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [12738.388719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12738.499534] alg: No test for adler32 (adler32-zlib) [12739.619038] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [12739.893996] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [12741.656387] Key type lgssc registered [12742.993630] Lustre: Echo OBD driver; http://www.lustre.org/ [12787.454903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12787.473449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12788.679619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12788.703755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12788.759230] Lustre: lustre-MDT0000: new disk, initializing [12788.837434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12788.856248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12793.494567] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12806.829429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12807.006230] Lustre: 403081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12807.130209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12807.134756] Lustre: Skipped 1 previous similar message [12807.238598] Lustre: lustre-MDT0001: new disk, initializing [12807.365532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12807.413908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12807.437213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12813.559856] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12825.950242] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12833.436128] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12841.792946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12842.204959] Lustre: lustre-OST0000: new disk, initializing [12842.213561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12842.222699] Lustre: 405052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12842.342503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12849.864440] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12850.839045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12850.852264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12851.063758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12861.122745] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12866.974557] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12867.371513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12873.072316] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12873.292924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12876.771817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12876.773221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12876.790753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12876.803152] Lustre: Skipped 1 previous similar message [12881.844950] Lustre: server umount lustre-OST0000 complete [12887.012251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12887.029568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12887.047360] Lustre: Skipped 2 previous similar messages [12892.129421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12892.140149] Lustre: Skipped 1 previous similar message [12892.966442] Lustre: server umount lustre-MDT0000 complete [12897.535860] LustreError: 403878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457584 with bad export cookie 5508709765432506032 [12897.552740] LustreError: 403878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12897.554228] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12898.333292] Lustre: server umount lustre-MDT0001 complete [12909.158358] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12911.431228] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:33:17 (1789457597) [12919.194089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12919.675472] LustreError: 407512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12919.694507] LustreError: 407512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12919.762164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12924.889356] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12924.898190] LustreError: 407513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12928.996368] LustreError: 407512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12934.115958] LustreError: 407513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12934.476606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12935.123954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12940.223417] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12950.464854] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12957.583824] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12965.589387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12965.905962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12972.678622] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [12981.427967] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12984.592834] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12992.251893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13003.253324] Lustre: server umount lustre-OST0000 complete [13009.889801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13009.891054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13009.915400] Lustre: Skipped 2 previous similar messages [13015.011286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13015.013796] Lustre: Skipped 2 previous similar messages [13018.276845] Lustre: server umount lustre-MDT0000 complete [13020.135330] LustreError: 407513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13020.152929] LustreError: 407513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13021.025486] Lustre: 401302:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789457692/real 1789457692] req@00000000d44e9a8a x1876382111049856/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789457708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13021.056734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13022.113579] Lustre: 401305:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789457693/real 1789457693] req@00000000cace23f1 x1876382111050112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789457709 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13023.387346] LustreError: 407494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457710 with bad export cookie 5508709765432506921 [13023.394041] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13023.404624] LustreError: 407494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13024.065285] Lustre: server umount lustre-MDT0001 complete [13036.827199] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13040.709947] Key type lgssc unregistered [13041.167960] LNet: 411240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13041.172787] LNetError: 411240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13042.220972] LNet: Removed LNI 192.168.202.134@tcp [13043.645608] Key type .llcrypt unregistered [13043.647262] Key type ._llcrypt unregistered [13073.135741] Key type ._llcrypt registered [13073.137736] Key type .llcrypt registered [13073.227926] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13073.463502] Key type .llcrypt unregistered [13073.465317] Key type ._llcrypt unregistered [13094.244500] Key type ._llcrypt registered [13094.246993] Key type .llcrypt registered [13094.400504] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [13095.403033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13095.485469] alg: No test for adler32 (adler32-zlib) [13096.597610] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [13096.869246] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [13098.555452] Key type lgssc registered [13099.877727] Lustre: Echo OBD driver; http://www.lustre.org/ [13108.426083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13108.444923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13109.975647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13114.952821] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13120.486153] LustreError: 413448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13120.501408] LustreError: 413448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13123.756446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13124.179038] LustreError: 413449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13124.218248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13129.700064] LustreError: 413448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13129.739135] LustreError: 413448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13131.060842] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13134.825573] LustreError: 413443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13134.839802] LustreError: 413443:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13134.876487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13138.052808] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13139.956404] LustreError: 413449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13139.983355] LustreError: 413449:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13146.023744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13146.513295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13153.158453] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13165.542589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13165.548251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13165.571477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13167.080575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13167.081336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.083257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13168.968503] Lustre: server umount lustre-OST0000 complete [13177.313311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13177.315785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13177.335773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13177.335784] Lustre: Skipped 1 previous similar message [13177.361367] Lustre: Skipped 1 previous similar message [13181.301836] Lustre: server umount lustre-MDT0000 complete [13182.435064] LustreError: 413444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13182.457464] LustreError: 413444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13185.708271] LustreError: 415557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457872 with bad export cookie 8590255942533773082 [13185.713129] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13185.721256] LustreError: 415557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13186.205445] Lustre: server umount lustre-MDT0001 complete [13198.901392] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13202.135086] Key type lgssc unregistered [13202.458735] LNet: 416597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13202.471596] LNetError: 416597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13202.509066] LNet: Removed LNI 192.168.202.134@tcp [13203.363655] Key type .llcrypt unregistered [13203.365136] Key type ._llcrypt unregistered [13230.052113] Key type ._llcrypt registered [13230.056815] Key type .llcrypt registered [13230.175847] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [13231.621565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13231.711557] alg: No test for adler32 (adler32-zlib) [13233.005655] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [13233.230053] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [13234.880298] Key type lgssc registered [13236.344745] Lustre: Echo OBD driver; http://www.lustre.org/ [13245.247803] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13245.271270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13260.834867] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13265.953502] LustreError: 417985:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [13292.640232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13300.167182] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13309.385752] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13317.112678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13317.680251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13317.801799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13322.305475] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13322.723277] LustreError: 418010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13322.743461] LustreError: 418010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13326.821808] LustreError: 419023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13326.855445] LustreError: 419023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13331.944534] LustreError: 419022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13332.785202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13333.335207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13339.764461] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13348.422881] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13355.661630] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13363.686436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13363.689885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13363.698378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13364.201838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13364.209231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13364.224829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13366.287727] Lustre: server umount lustre-OST0000 complete [13374.433931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13374.436302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13374.457848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13374.492782] Lustre: Skipped 2 previous similar messages [13377.892467] Lustre: server umount lustre-MDT0000 complete [13379.555023] LustreError: 419022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13379.570951] LustreError: 419022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13382.430311] LustreError: 417993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458069 with bad export cookie 16531175581319955138 [13382.437095] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13382.446603] LustreError: 417993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13383.233346] Lustre: server umount lustre-MDT0001 complete [13397.069932] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13401.177724] Key type lgssc unregistered [13401.623623] LNet: 421557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13401.629809] LNetError: 421557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13401.647403] LNet: Removed LNI 192.168.202.134@tcp [13402.769377] Key type .llcrypt unregistered [13402.772079] Key type ._llcrypt unregistered [13432.471922] Key type ._llcrypt registered [13432.474288] Key type .llcrypt registered [13432.697535] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [13433.980671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13434.016961] alg: No test for adler32 (adler32-zlib) [13435.158699] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [13435.460022] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [13437.133226] Key type lgssc registered [13438.568107] Lustre: Echo OBD driver; http://www.lustre.org/ [13451.188313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13451.246412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13452.880695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13459.329601] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13463.014162] LustreError: 422972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13463.049465] LustreError: 422972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13468.130531] LustreError: 422973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13469.971185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13470.383514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13476.045423] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13485.179797] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13491.807739] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13499.229810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13499.676863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13507.870815] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13516.124492] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13523.259486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13531.623080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13531.628473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13531.646085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13533.428967] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping) [13533.718996] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13537.894831] Lustre: server umount lustre-MDT0000 complete [13538.503944] LustreError: 424242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13540.704791] LustreError: 422953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.34@tcp arrived at 1789458227 with bad export cookie 1941623752828261524 [13540.735250] LustreError: 422953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13543.565384] LustreError: 422954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458230 with bad export cookie 1941623752828260859 [13543.571663] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13543.906251] LustreError: 424242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13543.912811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13543.915936] LustreError: 424242:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13543.926401] Lustre: Skipped 1 previous similar message [13543.932207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13543.935536] Lustre: Skipped 3 previous similar messages [13549.026244] LustreError: 422972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.032584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13549.038410] LustreError: 422972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13549.740483] Lustre: server umount lustre-OST0000 complete [13558.248236] LustreError: 422972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13558.278376] LustreError: 422972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13560.840018] Lustre: server umount lustre-MDT0001 complete [13576.852055] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13580.320890] Key type lgssc unregistered [13580.624348] LNet: 426686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13580.630059] LNetError: 426686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13581.672071] LNet: Removed LNI 192.168.202.134@tcp [13582.716479] Key type .llcrypt unregistered [13582.720653] Key type ._llcrypt unregistered [13611.381468] Key type ._llcrypt registered [13611.384666] Key type .llcrypt registered [13611.588664] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [13612.550941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13612.678770] alg: No test for adler32 (adler32-zlib) [13613.933565] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [13614.217785] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [13615.984217] Key type lgssc registered [13617.301819] Lustre: Echo OBD driver; http://www.lustre.org/ [13626.535481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13626.562298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13628.114168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13633.498556] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13633.511261] LustreError: 428099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13633.533416] LustreError: 428099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13638.625358] LustreError: 428100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13642.350184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13642.873263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13647.986341] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13657.151788] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13664.193768] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13671.489772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13671.984151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13673.013774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13679.177677] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13687.827929] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13693.412630] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13693.442359] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13693.455699] Lustre: Skipped 1 previous similar message [13695.429857] Lustre: server umount lustre-MDT0001 complete [13695.739709] LustreError: 429400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13698.540711] LustreError: 428099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13698.559733] LustreError: 428099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13703.651072] LustreError: 428100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13703.654061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13703.654244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13703.654249] Lustre: Skipped 1 previous similar message [13703.656922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13708.877371] Lustre: server umount lustre-OST0000 complete [13713.111396] Lustre: server umount lustre-MDT0000 complete [13733.687230] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13737.338671] Key type lgssc unregistered [13737.699583] LNet: 431796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13737.718069] LNetError: 431796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13737.743768] LNet: Removed LNI 192.168.202.134@tcp [13739.296146] Key type .llcrypt unregistered [13739.299733] Key type ._llcrypt unregistered [13768.338041] Key type ._llcrypt registered [13768.342982] Key type .llcrypt registered [13768.522605] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [13782.654901] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [13783.972706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13784.054772] alg: No test for adler32 (adler32-zlib) [13785.146536] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [13785.405208] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [13787.129764] Key type lgssc registered [13788.538686] Lustre: Echo OBD driver; http://www.lustre.org/ [13829.894309] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [13848.983056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13849.048429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13850.411482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13850.460169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13850.535461] Lustre: lustre-MDT0000: new disk, initializing [13850.681974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13850.698262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13855.294138] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13872.005502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13872.269872] Lustre: lustre-OST0000: new disk, initializing [13872.275837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13872.283194] Lustre: Skipped 1 previous similar message [13872.290903] Lustre: 436833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13872.380985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13880.038527] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13880.523833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13880.542937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13880.587989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13890.022589] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13908.592265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13908.774273] Lustre: 435815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13908.847108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13908.910057] Lustre: lustre-MDT0001: new disk, initializing [13909.055318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13909.093696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13909.106359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13914.004617] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [13924.031524] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13924.312920] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13929.989246] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13930.282886] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13937.637375] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13937.653514] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13937.667308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13938.921655] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.34@tcp (stopping) [13938.927665] Lustre: Skipped 1 previous similar message [13941.878823] Lustre: server umount lustre-MDT0001 complete [13947.875099] LustreError: 435822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13947.909786] LustreError: 435822:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13953.026738] LustreError: 435820:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13953.047159] LustreError: 435820:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13956.066923] LustreError: 438998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13956.085388] LustreError: 438998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13961.188573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13961.190411] LustreError: 435825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13961.211696] Lustre: Skipped 1 previous similar message [13961.215391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13966.111150] Lustre: server umount lustre-OST0000 complete [13966.307067] LustreError: 435826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13976.742268] Lustre: server umount lustre-MDT0000 complete [13992.313221] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [13995.679759] Key type lgssc unregistered [13996.070827] LNet: 440586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13996.097046] LNetError: 440586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13996.121966] LNet: Removed LNI 192.168.202.134@tcp [13997.028368] Key type .llcrypt unregistered [13997.031661] Key type ._llcrypt unregistered [14043.768761] Key type ._llcrypt registered [14043.770710] Key type .llcrypt registered [14043.969863] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [14044.359606] Key type .llcrypt unregistered [14044.361644] Key type ._llcrypt unregistered [14063.008929] Key type ._llcrypt registered [14063.011894] Key type .llcrypt registered [14063.117646] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [14064.443946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14064.577713] alg: No test for adler32 (adler32-zlib) [14065.625315] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [14066.088362] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [14067.784210] Key type lgssc registered [14069.172744] Lustre: Echo OBD driver; http://www.lustre.org/ [14100.341902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14100.380366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14101.733820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14101.810559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14101.918303] Lustre: lustre-MDT0000: new disk, initializing [14102.042302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14102.061448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14107.541774] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [14119.594598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14119.835439] Lustre: 443957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14119.931823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14119.941724] Lustre: Skipped 1 previous similar message [14120.056832] Lustre: lustre-MDT0001: new disk, initializing [14120.220192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14120.266029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14120.286367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14125.904569] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [14136.492482] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14143.958774] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14152.118635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14152.529047] Lustre: lustre-OST0000: new disk, initializing [14152.534867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14152.541502] Lustre: 445923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14152.660461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14158.926296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14158.937966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14159.134836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14160.254440] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [14171.461617] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14181.859576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14181.865213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14181.884624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14184.928924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14184.929641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14184.965662] Lustre: Skipped 2 previous similar messages [14186.863673] Lustre: server umount lustre-MDT0000 complete [14191.655639] LustreError: 443949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458878 with bad export cookie 5094117312015458419 [14191.665426] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14191.671520] LustreError: 443949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14192.525301] Lustre: server umount lustre-MDT0001 complete [14206.958512] Lustre: server umount lustre-OST0000 complete [14223.650594] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:55:09 (1789458909) [14232.993175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14233.660525] LustreError: 447955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14233.693637] LustreError: 447955:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14233.789888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14239.208764] LustreError: 447956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14239.545799] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [14243.301461] LustreError: 447955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14248.427855] LustreError: 447956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14250.186904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14250.733247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14256.365411] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [14265.970752] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14272.834877] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14293.616764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14294.015878] Lustre: Found index 0 for lustre-OST0000, updating log [14294.369101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14301.805387] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [14304.630467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14311.101867] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14320.103992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14320.110234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14320.115047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14320.163384] Lustre: Skipped 2 previous similar messages [14327.782183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14327.801088] Lustre: Skipped 4 previous similar messages [14332.384125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14333.070988] Lustre: server umount lustre-OST0000 complete [14353.379807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14353.396391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14353.417245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14354.915190] LNet: 1 local NIs in recovery (showing 1): 192.168.202.134@tcp [14365.164137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14365.444252] Lustre: server umount lustre-MDT0000 complete [14365.675588] LustreError: 448682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14365.716247] LustreError: 448682:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14369.399820] LustreError: 447937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459056 with bad export cookie 5094117312015459812 [14369.407957] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14369.416623] LustreError: 447937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14369.802723] Lustre: server umount lustre-MDT0001 complete [14380.821487] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:57:46 (1789459066) [14418.401376] LNet: 1 local NIs in recovery (showing 1): 192.168.202.134@tcp [15012.766873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15013.544707] LustreError: 454381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15013.648276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15018.019488] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [15018.980198] LustreError: 454382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15023.075183] LustreError: 454381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15028.786321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15029.378207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15033.857124] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [15040.790236] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15047.026787] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15069.841130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15070.297155] Lustre: Found index 0 for lustre-OST0000, updating log [15070.866254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15071.722490] LNet: 1 local NIs in recovery (showing 1): 192.168.202.134@tcp [15079.538282] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [15088.558109] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15092.150520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15092.158174] Lustre: Skipped 1 previous similar message [15092.162217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15092.167063] Lustre: Skipped 6 previous similar messages [15097.412511] Lustre: server umount lustre-OST0000 complete [15102.887614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15102.898121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15102.904278] Lustre: Skipped 1 previous similar message [15104.994402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15105.004706] Lustre: Skipped 4 previous similar messages [15109.100492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15109.106069] Lustre: Skipped 7 previous similar messages [15116.256449] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15116.476765] Lustre: server umount lustre-MDT0000 complete [15117.292814] LustreError: 456202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15117.318215] LustreError: 456202:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15120.294630] LustreError: 454361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459807 with bad export cookie 5094117312015460897 [15120.302728] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15120.310868] LustreError: 454361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15120.930416] Lustre: server umount lustre-MDT0001 complete [15132.131238] LNet: 1 local NIs in recovery (showing 1): 192.168.202.134@tcp [15154.252502] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [15165.634843] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [15207.983925] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:11:33 (1789459893) [15231.103776] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [15234.034262] Key type lgssc unregistered [15234.360876] LNet: 460997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15234.369757] LNetError: 460997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15332.713362] LNet: Removed LNI 192.168.202.134@tcp [15333.608416] Key type .llcrypt unregistered [15333.610071] Key type ._llcrypt unregistered [15351.084278] Key type ._llcrypt registered [15351.087423] Key type .llcrypt registered [15351.229120] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [15352.288865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15352.483602] alg: No test for adler32 (adler32-zlib) [15353.462534] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [15353.620913] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [15355.256204] Key type lgssc registered [15356.574708] Lustre: Echo OBD driver; http://www.lustre.org/ [15361.546730] LNet: Added LNI 192.168.202.134@tcp100 [8/256/0/180] [15362.417036] LNet: Added LNI 192.168.202.134@tcp101 [8/256/0/180] [15363.693805] LNet: Added LNI 192.168.202.134@tcp102 [8/256/0/180] [15364.476397] LNet: Added LNI 192.168.202.134@tcp103 [8/256/0/180] [15365.482239] LNet: Added LNI 192.168.202.134@tcp104 [8/256/0/180] [15366.743457] LNet: Added LNI 192.168.202.134@tcp105 [8/256/0/180] [15368.057495] LNet: Added LNI 192.168.202.134@tcp106 [8/256/0/180] [15369.142800] LNet: Added LNI 192.168.202.134@tcp107 [8/256/0/180] [15370.396730] LNet: Added LNI 192.168.202.134@tcp108 [8/256/0/180] [15371.372530] LNet: Added LNI 192.168.202.134@tcp109 [8/256/0/180] [15372.458779] LNet: Added LNI 192.168.202.134@tcp110 [8/256/0/180] [15373.459046] LNet: Added LNI 192.168.202.134@tcp111 [8/256/0/180] [15374.908422] LNet: Added LNI 192.168.202.134@tcp112 [8/256/0/180] [15376.222642] LNet: Added LNI 192.168.202.134@tcp113 [8/256/0/180] [15377.593959] LNet: Added LNI 192.168.202.134@tcp114 [8/256/0/180] [15378.911076] LNet: Added LNI 192.168.202.134@tcp115 [8/256/0/180] [15380.082033] LNet: Added LNI 192.168.202.134@tcp116 [8/256/0/180] [15381.279356] LNet: Added LNI 192.168.202.134@tcp117 [8/256/0/180] [15382.492035] LNet: Added LNI 192.168.202.134@tcp118 [8/256/0/180] [15383.958786] LNet: Added LNI 192.168.202.134@tcp119 [8/256/0/180] [15385.187537] LNet: Added LNI 192.168.202.134@tcp120 [8/256/0/180] [15386.134625] LNet: Added LNI 192.168.202.134@tcp121 [8/256/0/180] [15387.342096] LNet: Added LNI 192.168.202.134@tcp122 [8/256/0/180] [15388.671898] LNet: Added LNI 192.168.202.134@tcp123 [8/256/0/180] [15389.797988] LNet: Added LNI 192.168.202.134@tcp124 [8/256/0/180] [15390.911031] LNet: Added LNI 192.168.202.134@tcp125 [8/256/0/180] [15391.846298] LNet: Added LNI 192.168.202.134@tcp126 [8/256/0/180] [15393.023915] LNet: Added LNI 192.168.202.134@tcp127 [8/256/0/180] [15394.264157] LNet: Added LNI 192.168.202.134@tcp128 [8/256/0/180] [15395.820594] LNet: Added LNI 192.168.202.134@tcp129 [8/256/0/180] [15397.229464] LNet: Added LNI 192.168.202.134@tcp130 [8/256/0/180] [15398.810851] LNet: Added LNI 192.168.202.134@tcp131 [8/256/0/180] [15400.405480] LNet: Added LNI 192.168.202.134@tcp132 [8/256/0/180] [15402.120687] LNet: Added LNI 192.168.202.134@tcp133 [8/256/0/180] [15403.958848] LNet: Added LNI 192.168.202.134@tcp134 [8/256/0/180] [15405.368763] LNet: Added LNI 192.168.202.134@tcp135 [8/256/0/180] [15415.628688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15415.648571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15416.709352] Lustre: 464027:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.134@tcp: can't add NID 192.168.202.134@tcp130: rc = -75 [15416.759991] LNetError: 461482:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.134 [15416.777520] LNetError: Protocol error connecting to 192.168.202.134@tcp114 at host 192.168.202.134:988: is it running a compatible version of Lustre? [15416.787762] LNetError: Skipped 5 previous similar messages [15416.882675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15416.906770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15416.987611] Lustre: lustre-MDT0000: new disk, initializing [15417.081333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15417.098712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15417.122424] LNet: 8 local NIs in recovery (showing 8): 192.168.202.134@tcp122, 192.168.202.134@tcp123, 192.168.202.134@tcp124, 192.168.202.134@tcp125, 192.168.202.134@tcp128, 192.168.202.134@tcp129, 192.168.202.134@tcp126, 192.168.202.134@tcp127 [15419.233120] LNetError: 461481:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.134 [15419.251111] LNetError: 461481:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 19 previous similar messages [15419.268748] LNetError: Protocol error connecting to 192.168.202.134@tcp122 at host 192.168.202.134:988: is it running a compatible version of Lustre? [15419.280263] LNetError: Skipped 11 previous similar messages [15421.418912] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [15423.331281] LNetError: 464048:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.134 [15423.352744] LNetError: Protocol error connecting to 192.168.202.134@tcp122 at host 192.168.202.134:988: is it running a compatible version of Lustre? [15431.532346] LNetError: 464052:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.134 [15431.550161] LNetError: Protocol error connecting to 192.168.202.134@tcp122 at host 192.168.202.134:988: is it running a compatible version of Lustre? [15432.928443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15433.131914] Lustre: 464044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15433.310440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15433.319397] Lustre: Skipped 1 previous similar message [15433.542580] Lustre: lustre-MDT0001: new disk, initializing [15433.666331] Lustre: 464845:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.134@tcp: can't add NID 192.168.202.134@tcp130: rc = -75 [15433.674874] Lustre: 464845:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15433.922367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15433.993794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15434.018133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15439.485018] Lustre: 465050:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.134@tcp: can't add NID 192.168.202.134@tcp130: rc = -75 [15439.501541] Lustre: 465050:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15440.288992] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [15447.841723] LNetError: 464049:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.134 [15447.848270] LNetError: Protocol error connecting to 192.168.202.134@tcp114 at host 192.168.202.134:988: is it running a compatible version of Lustre? [15450.454057] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15456.034940] LNetError: 461483:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.134 [15456.062815] LNetError: 461483:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15456.088773] LNetError: Protocol error connecting to 192.168.202.134@tcp121 at host 192.168.202.134:988: is it running a compatible version of Lustre? [15456.121657] LNetError: Skipped 2 previous similar messages [15457.183980] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15466.217092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15466.578120] Lustre: lustre-OST0000: new disk, initializing [15466.587534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15466.663517] Lustre: 466017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15466.857415] Lustre: 466022:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.134@tcp: can't add NID 192.168.202.134@tcp130: rc = -75 [15466.866759] Lustre: 466022:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15466.947681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15473.441530] LNetError: 464049:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.134 [15473.452739] LNetError: 464049:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15473.459719] LNetError: Protocol error connecting to 192.168.202.134@tcp114 at host 192.168.202.134:988: is it running a compatible version of Lustre? [15473.472544] LNetError: Skipped 5 previous similar messages [15474.715747] Lustre: 466311:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.134@tcp: can't add NID 192.168.202.134@tcp130: rc = -75 [15474.733963] Lustre: 466311:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15474.813385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15474.837329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15475.083387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15476.102469] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [15477.664110] LNet: 1 local NIs in recovery (showing 1): 192.168.202.134@tcp122 [15488.164276] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15500.772529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15500.774634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15500.808795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15503.178508] Lustre: server umount lustre-OST0000 complete [15510.497738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15510.510755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15510.529150] Lustre: Skipped 1 previous similar message [15510.535142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15510.541297] Lustre: Skipped 1 previous similar message [15511.401341] LNetError: 464049:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.134 [15511.412755] LNetError: 464049:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15511.421254] LNetError: Protocol error connecting to 192.168.202.134@tcp114 at host 192.168.202.134:988: is it running a compatible version of Lustre? [15511.434462] LNetError: Skipped 8 previous similar messages [15514.441077] Lustre: server umount lustre-MDT0000 complete [15518.753081] LustreError: 464035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789460206 with bad export cookie 12710073243204714788 [15518.754589] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15518.773244] LustreError: 464035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15519.193310] Lustre: server umount lustre-MDT0001 complete [15532.089097] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [15536.260464] Key type lgssc unregistered [15536.636901] LNet: 467928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15536.645420] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15536.655569] LNet: Removed LNI 192.168.202.134@tcp [15537.696169] LNet: Removed LNI 192.168.202.134@tcp100 [15537.701190] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15537.707517] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15537.734777] LNet: Removed LNI 192.168.202.134@tcp101 [15538.784094] LNet: Removed LNI 192.168.202.134@tcp102 [15538.786129] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15538.808648] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15538.831449] LNet: Removed LNI 192.168.202.134@tcp103 [15539.875700] LNet: Removed LNI 192.168.202.134@tcp104 [15540.896193] LNet: Removed LNI 192.168.202.134@tcp105 [15540.898107] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15540.917811] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15540.926513] LNet: Removed LNI 192.168.202.134@tcp106 [15541.986091] LNet: Removed LNI 192.168.202.134@tcp107 [15543.009228] LNet: Removed LNI 192.168.202.134@tcp108 [15544.032162] LNet: Removed LNI 192.168.202.134@tcp109 [15545.056228] LNet: Removed LNI 192.168.202.134@tcp110 [15545.057738] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15545.067457] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15545.076639] LNet: Removed LNI 192.168.202.134@tcp111 [15546.144262] LNet: Removed LNI 192.168.202.134@tcp112 [15547.168458] LNet: Removed LNI 192.168.202.134@tcp113 [15548.192479] LNet: Removed LNI 192.168.202.134@tcp114 [15549.216722] LNet: Removed LNI 192.168.202.134@tcp115 [15550.240153] LNet: Removed LNI 192.168.202.134@tcp116 [15551.264144] LNet: Removed LNI 192.168.202.134@tcp117 [15552.288170] LNet: Removed LNI 192.168.202.134@tcp118 [15553.313245] LNet: Removed LNI 192.168.202.134@tcp119 [15553.321564] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15553.335488] LNetError: 467928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15553.344511] LNet: Removed LNI 192.168.202.134@tcp120 [15554.400960] LNet: Removed LNI 192.168.202.134@tcp121 [15554.405668] LNet: Removed LNI 192.168.202.134@tcp122 [15555.424198] LNet: Removed LNI 192.168.202.134@tcp123 [15556.448218] LNet: Removed LNI 192.168.202.134@tcp124 [15557.475609] LNet: Removed LNI 192.168.202.134@tcp125 [15558.497955] LNet: Removed LNI 192.168.202.134@tcp126 [15559.520330] LNet: Removed LNI 192.168.202.134@tcp127 [15560.544131] LNet: Removed LNI 192.168.202.134@tcp128 [15561.568223] LNet: Removed LNI 192.168.202.134@tcp129 [15561.578474] LNet: Removed LNI 192.168.202.134@tcp130 [15561.589679] LNet: Removed LNI 192.168.202.134@tcp131 [15561.597719] LNet: Removed LNI 192.168.202.134@tcp132 [15561.599252] LNet: Removed LNI 192.168.202.134@tcp133 [15561.611320] LNet: Removed LNI 192.168.202.134@tcp134 [15561.658730] LNet: Removed LNI 192.168.202.134@tcp135 [15562.906529] Key type .llcrypt unregistered [15562.908490] Key type ._llcrypt unregistered [15588.985680] Key type ._llcrypt registered [15588.987114] Key type .llcrypt registered [15589.175411] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [15605.579769] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [15606.994107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15607.257916] alg: No test for adler32 (adler32-zlib) [15608.346743] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [15608.595189] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [15610.298270] Key type lgssc registered [15611.666711] Lustre: Echo OBD driver; http://www.lustre.org/ [15650.431331] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:18:56 (1789460336) [15669.850399] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [15673.040492] Key type lgssc unregistered [15673.411680] LNet: 472013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15673.420053] LNetError: 472013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15673.438274] LNet: Removed LNI 192.168.202.134@tcp [15674.305066] Key type .llcrypt unregistered [15674.311178] Key type ._llcrypt unregistered [15691.962754] Key type ._llcrypt registered [15691.981431] Key type .llcrypt registered [15692.126820] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [15693.900593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15693.959673] alg: No test for adler32 (adler32-zlib) [15695.090632] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [15695.432677] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [15697.168500] Key type lgssc registered [15698.569416] Lustre: Echo OBD driver; http://www.lustre.org/ [15710.305173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15710.365311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15711.706913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15711.831674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15711.962907] Lustre: lustre-MDT0000: new disk, initializing [15712.097137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15712.119065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15718.533979] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [15731.590555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15731.757555] Lustre: 473226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15731.819860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15731.824202] Lustre: Skipped 1 previous similar message [15731.962379] Lustre: lustre-MDT0001: new disk, initializing [15732.080067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15732.130453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15732.147288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15737.437776] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [15747.443646] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15754.447872] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15763.216455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15763.710519] Lustre: lustre-OST0000: new disk, initializing [15763.718835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15763.728508] Lustre: 475194:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15763.828371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15771.392930] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [15773.811769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15773.830643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15773.999342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15782.695498] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15788.557635] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing lnet_if_list [15795.721813] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15795.784162] LNet: Added LNI 192.168.202.134@tcp10 [8/256/0/180] [15804.092450] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15804.146760] LNet: Added LNI 192.168.202.134@tcp11 [8/256/0/180] [15809.610770] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15809.647733] LNet: Added LNI 192.168.202.134@tcp12 [8/256/0/180] [15815.320830] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15815.341562] LNet: Added LNI 192.168.202.134@tcp13 [8/256/0/180] [15821.109553] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15821.137915] LNet: Added LNI 192.168.202.134@tcp14 [8/256/0/180] [15827.110544] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15827.135400] LNet: Added LNI 192.168.202.134@tcp15 [8/256/0/180] [15833.187879] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15833.231990] LNet: Added LNI 192.168.202.134@tcp16 [8/256/0/180] [15840.359466] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15840.390608] LNet: Added LNI 192.168.202.134@tcp17 [8/256/0/180] [15845.997435] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15846.023813] LNet: Added LNI 192.168.202.134@tcp18 [8/256/0/180] [15852.733911] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15852.777477] LNet: Added LNI 192.168.202.134@tcp19 [8/256/0/180] [15859.356946] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15859.380637] LNet: Added LNI 192.168.202.134@tcp20 [8/256/0/180] [15865.587662] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15865.614770] LNet: Added LNI 192.168.202.134@tcp21 [8/256/0/180] [15872.036587] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15872.093333] LNet: Added LNI 192.168.202.134@tcp22 [8/256/0/180] [15878.885891] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15878.910970] LNet: Added LNI 192.168.202.134@tcp23 [8/256/0/180] [15885.746955] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15885.827285] LNet: Added LNI 192.168.202.134@tcp24 [8/256/0/180] [15893.290317] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15893.329291] LNet: Added LNI 192.168.202.134@tcp25 [8/256/0/180] [15899.543179] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15899.561089] LNet: Added LNI 192.168.202.134@tcp26 [8/256/0/180] [15906.712245] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15906.746445] LNet: Added LNI 192.168.202.134@tcp27 [8/256/0/180] [15914.284860] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15914.333885] LNet: Added LNI 192.168.202.134@tcp28 [8/256/0/180] [15921.778916] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15921.793097] LNet: Added LNI 192.168.202.134@tcp29 [8/256/0/180] [15929.631923] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15929.666259] LNet: Added LNI 192.168.202.134@tcp30 [8/256/0/180] [15937.130903] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15937.160787] LNet: Added LNI 192.168.202.134@tcp31 [8/256/0/180] [15944.257275] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15944.303036] LNet: Added LNI 192.168.202.134@tcp32 [8/256/0/180] [15951.361575] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15951.409263] LNet: Added LNI 192.168.202.134@tcp33 [8/256/0/180] [15957.813902] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15957.834033] LNet: Added LNI 192.168.202.134@tcp34 [8/256/0/180] [15964.470427] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15964.502714] LNet: Added LNI 192.168.202.134@tcp35 [8/256/0/180] [15971.440533] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15971.477932] LNet: Added LNI 192.168.202.134@tcp36 [8/256/0/180] [15978.587729] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15978.612790] LNet: Added LNI 192.168.202.134@tcp37 [8/256/0/180] [15987.189349] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15987.240763] LNet: Added LNI 192.168.202.134@tcp38 [8/256/0/180] [15995.087293] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15995.132485] LNet: Added LNI 192.168.202.134@tcp39 [8/256/0/180] [16003.405262] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16003.449485] LNet: Added LNI 192.168.202.134@tcp40 [8/256/0/180] [16011.986636] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16012.044400] LNet: Added LNI 192.168.202.134@tcp41 [8/256/0/180] [16021.995245] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16022.047041] LNet: Added LNI 192.168.202.134@tcp42 [8/256/0/180] [16031.517891] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16031.560707] LNet: Added LNI 192.168.202.134@tcp43 [8/256/0/180] [16040.642137] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16040.671961] LNet: Added LNI 192.168.202.134@tcp44 [8/256/0/180] [16051.037419] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16051.069298] LNet: Added LNI 192.168.202.134@tcp45 [8/256/0/180] [16060.310885] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16060.336879] LNet: Added LNI 192.168.202.134@tcp46 [8/256/0/180] [16070.533608] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16070.562934] LNet: Added LNI 192.168.202.134@tcp47 [8/256/0/180] [16079.704700] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16079.751343] LNet: Added LNI 192.168.202.134@tcp48 [8/256/0/180] [16089.260790] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16089.304503] LNet: Added LNI 192.168.202.134@tcp49 [8/256/0/180] [16098.976448] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16099.006429] LNet: Added LNI 192.168.202.134@tcp50 [8/256/0/180] [16121.328627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16121.341057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16121.367731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16122.341200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16122.349886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16122.364742] Lustre: Skipped 2 previous similar messages [16127.208719] Lustre: server umount lustre-MDT0000 complete [16131.298791] LustreError: 473217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789460818 with bad export cookie 980872878142165974 [16131.301089] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16131.322060] LustreError: 473217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16132.007970] Lustre: server umount lustre-MDT0001 complete [16147.744180] Lustre: 472497:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789460819/real 1789460819] req@00000000407b7ac3 x1876385209700864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789460835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16147.760136] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16147.779376] Lustre: Skipped 1 previous similar message [16152.032139] Lustre: 472495:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789460823/real 1789460823] req@00000000cffefbab x1876385209700992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789460839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16153.057229] Lustre: 472495:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789460824/real 1789460824] req@00000000413437ee x1876385209701120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789460840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16153.622762] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [16158.176117] Lustre: 472498:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789460829/real 1789460829] req@00000000e09b31a4 x1876385209701376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789460845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16162.483788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16163.361674] Lustre: 472498:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789460834/real 1789460834] req@00000000ed9ee402 x1876385209701504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789460850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16177.812890] LustreError: 475188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16177.845512] LustreError: 475188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16177.897028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16178.913784] LustreError: 475189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16182.952894] LustreError: 475188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16183.128192] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16188.392971] LustreError: 475189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16193.515362] LustreError: 475188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16193.541352] LustreError: 475188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16193.862804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16194.128508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16198.711638] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16199.160759] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16202.249055] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16206.788271] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16213.466355] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16234.465291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16234.481617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16234.510211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16238.050469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16238.062713] Lustre: Skipped 2 previous similar messages [16240.419485] Lustre: server umount lustre-OST0000 complete [16245.218644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16245.225474] LustreError: Skipped 1 previous similar message [16245.231734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16245.255377] Lustre: Skipped 1 previous similar message [16245.273915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16251.383134] Lustre: server umount lustre-MDT0000 complete [16253.441122] LustreError: 499439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16253.469090] LustreError: 499439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16255.325445] LustreError: 473218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789460942 with bad export cookie 980872878142168557 [16255.333223] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16255.338743] LustreError: 473218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16255.675460] Lustre: server umount lustre-MDT0001 complete [16266.837396] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [16269.761249] Key type lgssc unregistered [16270.103180] LNet: 502240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16270.113785] LNetError: 502240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16270.119892] LNet: Removed LNI 192.168.202.134@tcp [16270.124916] LNet: Removed LNI 192.168.202.134@tcp10 [16270.129638] LNet: Removed LNI 192.168.202.134@tcp11 [16270.132548] LNet: Removed LNI 192.168.202.134@tcp12 [16270.135323] LNet: Removed LNI 192.168.202.134@tcp13 [16270.138590] LNet: Removed LNI 192.168.202.134@tcp14 [16270.142727] LNet: Removed LNI 192.168.202.134@tcp15 [16270.147584] LNet: Removed LNI 192.168.202.134@tcp16 [16270.149472] LNet: Removed LNI 192.168.202.134@tcp17 [16270.160220] LNet: Removed LNI 192.168.202.134@tcp18 [16270.166138] LNet: Removed LNI 192.168.202.134@tcp19 [16270.172123] LNet: Removed LNI 192.168.202.134@tcp20 [16270.178922] LNet: Removed LNI 192.168.202.134@tcp21 [16270.184785] LNet: Removed LNI 192.168.202.134@tcp22 [16270.186578] LNet: Removed LNI 192.168.202.134@tcp23 [16270.190248] LNet: Removed LNI 192.168.202.134@tcp24 [16270.193824] LNet: Removed LNI 192.168.202.134@tcp25 [16270.195957] LNet: Removed LNI 192.168.202.134@tcp26 [16270.198277] LNet: Removed LNI 192.168.202.134@tcp27 [16270.206374] LNet: Removed LNI 192.168.202.134@tcp28 [16270.210298] LNet: Removed LNI 192.168.202.134@tcp29 [16270.213490] LNet: Removed LNI 192.168.202.134@tcp30 [16270.219441] LNet: Removed LNI 192.168.202.134@tcp31 [16270.223221] LNet: Removed LNI 192.168.202.134@tcp32 [16270.228463] LNet: Removed LNI 192.168.202.134@tcp33 [16270.231637] LNet: Removed LNI 192.168.202.134@tcp34 [16270.238224] LNet: Removed LNI 192.168.202.134@tcp35 [16270.241981] LNet: Removed LNI 192.168.202.134@tcp36 [16270.245438] LNet: Removed LNI 192.168.202.134@tcp37 [16270.249620] LNet: Removed LNI 192.168.202.134@tcp38 [16270.252820] LNet: Removed LNI 192.168.202.134@tcp39 [16270.259383] LNet: Removed LNI 192.168.202.134@tcp40 [16270.262928] LNet: Removed LNI 192.168.202.134@tcp41 [16270.266456] LNet: Removed LNI 192.168.202.134@tcp42 [16270.268966] LNet: Removed LNI 192.168.202.134@tcp43 [16270.272366] LNet: Removed LNI 192.168.202.134@tcp44 [16270.275428] LNet: Removed LNI 192.168.202.134@tcp45 [16270.278369] LNet: Removed LNI 192.168.202.134@tcp46 [16270.281467] LNet: Removed LNI 192.168.202.134@tcp47 [16270.284362] LNet: Removed LNI 192.168.202.134@tcp48 [16270.287234] LNet: Removed LNI 192.168.202.134@tcp49 [16270.298678] LNet: Removed LNI 192.168.202.134@tcp50 [16271.245925] Key type .llcrypt unregistered [16271.252456] Key type ._llcrypt unregistered [16293.422380] Key type ._llcrypt registered [16293.426971] Key type .llcrypt registered [16293.556465] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [16310.841124] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [16312.303453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16312.454330] alg: No test for adler32 (adler32-zlib) [16313.542329] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [16313.771320] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [16315.427148] Key type lgssc registered [16316.592559] Lustre: Echo OBD driver; http://www.lustre.org/ [16354.558111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16354.577548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16355.926850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16355.965827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16356.098215] Lustre: lustre-MDT0000: new disk, initializing [16356.250556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16356.294419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16361.692664] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16372.378402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16372.571283] Lustre: 505437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16372.646286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16372.654818] Lustre: Skipped 1 previous similar message [16372.744445] Lustre: lustre-MDT0001: new disk, initializing [16372.847508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16372.901222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16372.920876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16378.595738] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16389.231991] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16395.170154] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16404.025426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16404.209542] Lustre: *** cfs_fail_loc=90f, val=0*** [16404.263854] Lustre: lustre-OST0000: new disk, initializing [16404.266318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16404.270240] Lustre: 507452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16404.345657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16411.883619] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16412.254177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16412.261390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16412.366911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16421.563211] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16433.126706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16433.129585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16433.134130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16438.244494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16438.246559] Lustre: Skipped 1 previous similar message [16442.336159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16442.566213] Lustre: server umount lustre-OST0000 complete [16448.485312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16448.496951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16448.505444] Lustre: Skipped 2 previous similar messages [16448.523241] Lustre: Skipped 1 previous similar message [16453.603721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16453.612978] Lustre: Skipped 1 previous similar message [16458.721519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16458.735188] Lustre: Skipped 1 previous similar message [16461.280139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16461.679778] Lustre: server umount lustre-MDT0000 complete [16466.542731] LustreError: 505431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789461153 with bad export cookie 9242656007846404594 [16466.550601] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16466.552954] LustreError: 505431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16466.935164] Lustre: server umount lustre-MDT0001 complete [16476.570489] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [16478.750944] Key type lgssc unregistered [16479.033715] LNet: 509362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16479.037448] LNetError: 509362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16479.052730] LNet: Removed LNI 192.168.202.134@tcp [16479.860511] Key type .llcrypt unregistered [16479.862326] Key type ._llcrypt unregistered [16499.928330] Key type ._llcrypt registered [16499.933850] Key type .llcrypt registered [16500.068605] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [16501.059765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16501.075036] alg: No test for adler32 (adler32-zlib) [16502.354843] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [16502.571688] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [16504.240254] Key type lgssc registered [16505.384862] Lustre: Echo OBD driver; http://www.lustre.org/ [16513.552508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16513.581854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16514.968856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16519.893517] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16525.286447] LustreError: 510629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16525.312492] LustreError: 510629:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16528.916680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16529.311740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16534.039882] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16541.476394] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16547.611119] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16554.538154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16554.847024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16561.312791] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16568.938156] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16587.657307] loop0: detected capacity change from 0 to 204800000 [16588.068307] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16588.075724] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16588.085262] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16597.017959] loop0: detected capacity change from 0 to 204800000 [16597.447209] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16597.474394] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16597.489426] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16603.038368] loop0: detected capacity change from 0 to 204800000 [16608.811603] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16608.945624] Lustre: fs15246-OST0000: new disk, initializing [16608.949447] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16608.953958] Lustre: 514221:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16608.997940] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16615.623918] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16625.306432] loop1: detected capacity change from 0 to 204800000 [16631.373749] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16631.548473] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16631.607749] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16631.703104] Lustre: fs15246-MDT0000: new disk, initializing [16631.854622] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16631.883703] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16631.967861] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16633.001384] Lustre: 515343:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000001a6e4ab x1876386055767168/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789461336 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16636.853212] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16638.124316] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16638.134366] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16638.155641] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16663.034877] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16663.059648] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16664.940977] Lustre: server umount fs15246-MDT0000 complete [16668.140702] LustreError: 510624:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16673.170650] Lustre: server umount fs15246-OST0000 complete [16688.099934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16688.108169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16688.131127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16693.749310] LustreError: 516047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16693.769970] LustreError: 516047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16694.150066] Lustre: server umount lustre-OST0000 complete [16703.458644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16703.470139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16703.486711] Lustre: Skipped 1 previous similar message [16703.507350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16703.512689] Lustre: Skipped 1 previous similar message [16705.785864] Lustre: server umount lustre-MDT0000 complete [16708.857064] LustreError: 510610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789461396 with bad export cookie 2432371833698413486 [16708.866812] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16708.867819] LustreError: 510610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16709.339621] Lustre: server umount lustre-MDT0001 complete [16721.497566] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [16724.647561] Key type lgssc unregistered [16725.002585] LNet: 518441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16725.012439] LNetError: 518441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16725.031542] LNet: Removed LNI 192.168.202.134@tcp [16726.102720] Key type .llcrypt unregistered [16726.105865] Key type ._llcrypt unregistered [16746.732820] Key type ._llcrypt registered [16746.734864] Key type .llcrypt registered [16746.878486] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [16748.471360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16748.498531] alg: No test for adler32 (adler32-zlib) [16749.526573] Lustre: Lustre: Build Version: 2.17.58_39_ge07ba0b [16749.760330] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [16751.480230] Key type lgssc registered [16752.787550] Lustre: Echo OBD driver; http://www.lustre.org/ [16797.326835] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [16809.189934] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [16850.503134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16850.524822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16851.748168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16851.768969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16851.864993] Lustre: lustre-MDT0000: new disk, initializing [16851.936497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16851.954377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16858.955559] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16872.650208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16872.855226] Lustre: 522907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16872.997182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16873.003295] Lustre: Skipped 1 previous similar message [16873.114837] Lustre: lustre-MDT0001: new disk, initializing [16873.250818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16873.326439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16873.344861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16878.760565] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16889.637627] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16897.149531] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16906.292583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16906.761963] Lustre: lustre-OST0000: new disk, initializing [16906.769808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16906.786830] Lustre: 524876:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16906.920423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16912.471372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16912.489282] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16912.685965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16915.678964] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [16929.439108] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16935.252735] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16935.396496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16939.027237] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16939.229552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16941.550093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16941.557909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16941.565737] Lustre: Skipped 1 previous similar message [16941.578033] Lustre: Skipped 1 previous similar message [16946.670315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16946.702494] Lustre: Skipped 1 previous similar message [16951.782994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16951.799549] Lustre: Skipped 1 previous similar message [16955.360219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16955.732276] Lustre: server umount lustre-OST0000 complete [16962.018056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16962.051899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16962.056476] Lustre: Skipped 1 previous similar message [16967.142256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16967.153177] Lustre: Skipped 1 previous similar message [16967.883143] Lustre: server umount lustre-MDT0000 complete [16972.336148] LustreError: 522898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789461659 with bad export cookie 10006576565103996136 [16972.340202] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16972.369828] LustreError: 522898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16972.790316] Lustre: server umount lustre-MDT0001 complete [17003.908906] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [17015.541952] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [17051.461839] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16798 sec ========= 04:42:16 (1789461736) [17054.248969] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:42:19 (1789461739) === [17058.922899] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:42:24 (1789461744) === [17105.155592] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [17107.826641] Key type lgssc unregistered [17108.118893] LNet: 531524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17108.123799] LNetError: 531524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17108.145707] LNet: Removed LNI 192.168.202.134@tcp [17109.200427] Key type .llcrypt unregistered [17109.204537] Key type ._llcrypt unregistered